Horrible denial of service bug in autmount 5

Horrible denial of service bug in autmount 5

[Anton Altaparmakov]

Hi,

The autofs 5.0.2 package that comes with opensuse 10.3 has a nasty  
denial of service attack in the automount daemon.  I can only assume  
that the code comes from the actual autofs source rather than being  
introduced by opensuse but I could be wrong (I haven't checked).

The bug is that automount searches /proc/*/cmdline for a substring  
that matches "automount" and refuses to run if it finds such a thing.

So any user that just does:

cat > automount.c <<EOD
int main(void)
{
	sleep(10000);
	return 0;
}
EOD
gcc -o automount.c
export PATH=.;$PATH
automount

And now no-one can run the real automount including root!

Even if this was not a DoS waiting to happen, why do you have this  
check in the code?  There is no reason whatsoever to try and restrict  
people from running multiple instances of the automount process...

We in fact run one automount instance for each logged in user on our  
Linux distribution for Cambridge University. - We now have to play  
silly buggers with running automount in such a way as to replace its  
argv[0] with a different string so we can run multiple instances.

But that still leaves the DoS attack that any user can run a program  
as above and no-one else will be able to log in any more as the  
automount process will find the literal string "automount" from the  
user's executable...

So we would really like the complete abomination that is autofs/daemon/ 
automount.c::is_automount_running() thrown away or at least made  
optional with a command line option if you insist on having it, pretty  
please with sugar on top?

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/

Re: Horrible denial of service bug in autmount 5

[Ian Kent]

On Fri, 2008-06-20 at 14:24 +0100, Anton Altaparmakov wrote:
> Hi,
> 
> The autofs 5.0.2 package that comes with opensuse 10.3 has a nasty  
> denial of service attack in the automount daemon.  I can only assume  
> that the code comes from the actual autofs source rather than being  
> introduced by opensuse but I could be wrong (I haven't checked).
> 
> The bug is that automount searches /proc/*/cmdline for a substring  
> that matches "automount" and refuses to run if it finds such a thing.
> 
> So any user that just does:
> 
> cat > automount.c <<EOD
> int main(void)
> {
> 	sleep(10000);
> 	return 0;
> }
> EOD
> gcc -o automount.c
> export PATH=.;$PATH
> automount
> 
> And now no-one can run the real automount including root!
> 
> Even if this was not a DoS waiting to happen, why do you have this  
> check in the code?  There is no reason whatsoever to try and restrict  
> people from running multiple instances of the automount process...

Really, the fact that you think no-one will ever attempt to run
automount again, perhaps by accidentally starting the application when
it is already running, using the same configuration shows you haven't
really thought about this issue.
  
> 
> We in fact run one automount instance for each logged in user on our  
> Linux distribution for Cambridge University. - We now have to play  
> silly buggers with running automount in such a way as to replace its  
> argv[0] with a different string so we can run multiple instances.
> 
> But that still leaves the DoS attack that any user can run a program  
> as above and no-one else will be able to log in any more as the  
> automount process will find the literal string "automount" from the  
> user's executable...
> 
> So we would really like the complete abomination that is autofs/daemon/ 
> automount.c::is_automount_running() thrown away or at least made  
> optional with a command line option if you insist on having it, pretty  
> please with sugar on top?

The tone of your mail is lousy, given that your asking for something and
haven't offered a patch to support your request and haven't really
thought about the issue and haven't even offered any suggestions about
alternative approaches. 

Ian

Re: Horrible denial of service bug in autmount 5

[Anton Altaparmakov]

Hi,

[Note: Corrected hpa's email address - I didn't notice it was still  
the transmeta one...  Oops.]

On 20 Jun 2008, at 14:44, Ian Kent wrote:
> On Fri, 2008-06-20 at 14:24 +0100, Anton Altaparmakov wrote:
>> Hi,
>>
>> The autofs 5.0.2 package that comes with opensuse 10.3 has a nasty
>> denial of service attack in the automount daemon.  I can only assume
>> that the code comes from the actual autofs source rather than being
>> introduced by opensuse but I could be wrong (I haven't checked).
>>
>> The bug is that automount searches /proc/*/cmdline for a substring
>> that matches "automount" and refuses to run if it finds such a thing.
>>
>> So any user that just does:
>>
>> cat > automount.c <<EOD
>> int main(void)
>> {
>> 	sleep(10000);
>> 	return 0;
>> }
>> EOD
>> gcc -o automount.c
>> export PATH=.;$PATH
>> automount
>>
>> And now no-one can run the real automount including root!
>>
>> Even if this was not a DoS waiting to happen, why do you have this
>> check in the code?  There is no reason whatsoever to try and restrict
>> people from running multiple instances of the automount process...
>
> Really, the fact that you think no-one will ever attempt to run
> automount again, perhaps by accidentally starting the application when
> it is already running, using the same configuration shows you haven't
> really thought about this issue.

And so what?  The previous automount binary did not do this check  
either.  Lots of people use multiple invocations of the automounter in  
all sorts of scenarios and you are gratuitously breaking all of those  
applications or making them rather annoying.

But most importantly you are creating a DoS in your application  
because ANY user can run the example code I showed in my original post  
and thus stop the real automount process from running.

I am sorry but this is Unix.  You are not allowed to protect people  
from shooting themselves in the foot.  That's what init scripts are  
for!  I have never seen such a stupid check in a daemon before!

>> We in fact run one automount instance for each logged in user on our
>> Linux distribution for Cambridge University. - We now have to play
>> silly buggers with running automount in such a way as to replace its
>> argv[0] with a different string so we can run multiple instances.
>>
>> But that still leaves the DoS attack that any user can run a program
>> as above and no-one else will be able to log in any more as the
>> automount process will find the literal string "automount" from the
>> user's executable...
>>
>> So we would really like the complete abomination that is autofs/ 
>> daemon/
>> automount.c::is_automount_running() thrown away or at least made
>> optional with a command line option if you insist on having it,  
>> pretty
>> please with sugar on top?
>
> The tone of your mail is lousy, given that your asking for something  
> and
> haven't offered a patch to support your request and haven't really
> thought about the issue and haven't even offered any suggestions about
> alternative approaches.


You need a patch to delete a few lines of code?  Wow!  I expected you  
to be able to do it yourself!

And I did suggest two alternative approaches...  Reread my above  
paragraph.  It clearly states that completely removing the  
is_automount_running() function and its call site is one approach and  
another one is to provide a command line option to bypass execution of  
this code but that does not solve the DoS issue so only removal of the  
code is a real solution.

Looking for literal strings in /proc is nothing short of stupid!  I am  
sorry for you and your users if you can't see that...

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/

Re: Horrible denial of service bug in autmount 5

[Ian Kent]

On Fri, 2008-06-20 at 15:01 +0100, Anton Altaparmakov wrote:
> Hi,
> 
> [Note: Corrected hpa's email address - I didn't notice it was still  
> the transmeta one...  Oops.]
> 
> On 20 Jun 2008, at 14:44, Ian Kent wrote:
> > On Fri, 2008-06-20 at 14:24 +0100, Anton Altaparmakov wrote:
> >> Hi,
> >>
> >> The autofs 5.0.2 package that comes with opensuse 10.3 has a nasty
> >> denial of service attack in the automount daemon.  I can only assume
> >> that the code comes from the actual autofs source rather than being
> >> introduced by opensuse but I could be wrong (I haven't checked).
> >>
> >> The bug is that automount searches /proc/*/cmdline for a substring
> >> that matches "automount" and refuses to run if it finds such a thing.
> >>
> >> So any user that just does:
> >>
> >> cat > automount.c <<EOD
> >> int main(void)
> >> {
> >> 	sleep(10000);
> >> 	return 0;
> >> }
> >> EOD
> >> gcc -o automount.c
> >> export PATH=.;$PATH
> >> automount
> >>
> >> And now no-one can run the real automount including root!
> >>
> >> Even if this was not a DoS waiting to happen, why do you have this
> >> check in the code?  There is no reason whatsoever to try and restrict
> >> people from running multiple instances of the automount process...
> >
> > Really, the fact that you think no-one will ever attempt to run
> > automount again, perhaps by accidentally starting the application when
> > it is already running, using the same configuration shows you haven't
> > really thought about this issue.
> 
> And so what?  The previous automount binary did not do this check  
> either.  Lots of people use multiple invocations of the automounter in  
> all sorts of scenarios and you are gratuitously breaking all of those  
> applications or making them rather annoying.
> 
> But most importantly you are creating a DoS in your application  
> because ANY user can run the example code I showed in my original post  
> and thus stop the real automount process from running.
> 
> I am sorry but this is Unix.  You are not allowed to protect people  
> from shooting themselves in the foot.  That's what init scripts are  
> for!  I have never seen such a stupid check in a daemon before!

Ummm ... rubbish ... and rubbish and yeah the check is inconvenient but
not as bad as you claim.

I'm allowed to try and protect people from doing things that aren't
sensible if I wish to, "it is Unix" as you say.
 
> 
> >> We in fact run one automount instance for each logged in user on our
> >> Linux distribution for Cambridge University. - We now have to play
> >> silly buggers with running automount in such a way as to replace its
> >> argv[0] with a different string so we can run multiple instances.
> >>
> >> But that still leaves the DoS attack that any user can run a program
> >> as above and no-one else will be able to log in any more as the
> >> automount process will find the literal string "automount" from the
> >> user's executable...
> >>
> >> So we would really like the complete abomination that is autofs/ 
> >> daemon/
> >> automount.c::is_automount_running() thrown away or at least made
> >> optional with a command line option if you insist on having it,  
> >> pretty
> >> please with sugar on top?
> >
> > The tone of your mail is lousy, given that your asking for something  
> > and
> > haven't offered a patch to support your request and haven't really
> > thought about the issue and haven't even offered any suggestions about
> > alternative approaches.
> 
> 
> You need a patch to delete a few lines of code?  Wow!  I expected you  
> to be able to do it yourself!
> 
> And I did suggest two alternative approaches...  Reread my above  
> paragraph.  It clearly states that completely removing the  
> is_automount_running() function and its call site is one approach and  
> another one is to provide a command line option to bypass execution of  
> this code but that does not solve the DoS issue so only removal of the  
> code is a real solution.

Removing the function isn't a solution that I'm willing to accept, in
fact it isn't a solution at all, and since it's so simple you can easily
do it yourself and spend the time you save on coming up with a sensible
suggestion.

> 
> Looking for literal strings in /proc is nothing short of stupid!  I am  
> sorry for you and your users if you can't see that...

I never said that you point was not well taken.

I said I didn't like your tone, given that you were asking for something
and that continues to be the case. If you think that posting using this
tone will get priority for fixing this then you are sadly mistaken.

Ian

Re: Horrible denial of service bug in autmount 5

[Anton Altaparmakov]

Hi,

On 20 Jun 2008, at 16:04, Ian Kent wrote:
> On Fri, 2008-06-20 at 15:01 +0100, Anton Altaparmakov wrote:
>> On 20 Jun 2008, at 14:44, Ian Kent wrote:
>>> On Fri, 2008-06-20 at 14:24 +0100, Anton Altaparmakov wrote:
>>>> The autofs 5.0.2 package that comes with opensuse 10.3 has a nasty
>>>> denial of service attack in the automount daemon.  I can only  
>>>> assume
>>>> that the code comes from the actual autofs source rather than being
>>>> introduced by opensuse but I could be wrong (I haven't checked).
>>>>
>>>> The bug is that automount searches /proc/*/cmdline for a substring
>>>> that matches "automount" and refuses to run if it finds such a  
>>>> thing.
>>>>
>>>> So any user that just does:
>>>>
>>>> cat > automount.c <<EOD
>>>> int main(void)
>>>> {
>>>> 	sleep(10000);
>>>> 	return 0;
>>>> }
>>>> EOD
>>>> gcc -o automount.c
>>>> export PATH=.;$PATH
>>>> automount
>>>>
>>>> And now no-one can run the real automount including root!
>>>>
>>>> Even if this was not a DoS waiting to happen, why do you have this
>>>> check in the code?  There is no reason whatsoever to try and  
>>>> restrict
>>>> people from running multiple instances of the automount process...
>>>
>>> Really, the fact that you think no-one will ever attempt to run
>>> automount again, perhaps by accidentally starting the application  
>>> when
>>> it is already running, using the same configuration shows you  
>>> haven't
>>> really thought about this issue.
>>
>> And so what?  The previous automount binary did not do this check
>> either.  Lots of people use multiple invocations of the automounter  
>> in
>> all sorts of scenarios and you are gratuitously breaking all of those
>> applications or making them rather annoying.
>>
>> But most importantly you are creating a DoS in your application
>> because ANY user can run the example code I showed in my original  
>> post
>> and thus stop the real automount process from running.
>>
>> I am sorry but this is Unix.  You are not allowed to protect people
>> from shooting themselves in the foot.  That's what init scripts are
>> for!  I have never seen such a stupid check in a daemon before!
>
> Ummm ... rubbish ... and rubbish and yeah the check is inconvenient  
> but
> not as bad as you claim.

Sorry but it is worse than you make out, too.  Why don't you try  
running the snippet I posted as a normal user and after that try to  
restart the automounter for example?  You will notice it will fail  
telling you that it is already running even though it is not.

If the start (or restart) had been done automatically as part of  
installing an updated rpm then suddenly you have a potentially broken  
server (think of automounted home directories for example which is  
what we have).

And there are plenty of people who set their servers to update  
automatically during the night.  (And no I am not one of them!)

> I'm allowed to try and protect people from doing things that aren't
> sensible if I wish to, "it is Unix" as you say.

Yes, sorry.  I meant to write that it is against Unix Philosophy not  
that you are not allowed to do it.  (Brain to keyboard translation  
error. - It is Friday!)

>>>> We in fact run one automount instance for each logged in user on  
>>>> our
>>>> Linux distribution for Cambridge University. - We now have to play
>>>> silly buggers with running automount in such a way as to replace  
>>>> its
>>>> argv[0] with a different string so we can run multiple instances.
>>>>
>>>> But that still leaves the DoS attack that any user can run a  
>>>> program
>>>> as above and no-one else will be able to log in any more as the
>>>> automount process will find the literal string "automount" from the
>>>> user's executable...
>>>>
>>>> So we would really like the complete abomination that is autofs/
>>>> daemon/
>>>> automount.c::is_automount_running() thrown away or at least made
>>>> optional with a command line option if you insist on having it,
>>>> pretty
>>>> please with sugar on top?
>>>
>>> The tone of your mail is lousy, given that your asking for something
>>> and
>>> haven't offered a patch to support your request and haven't really
>>> thought about the issue and haven't even offered any suggestions  
>>> about
>>> alternative approaches.
>>
>> You need a patch to delete a few lines of code?  Wow!  I expected you
>> to be able to do it yourself!
>>
>> And I did suggest two alternative approaches...  Reread my above
>> paragraph.  It clearly states that completely removing the
>> is_automount_running() function and its call site is one approach and
>> another one is to provide a command line option to bypass execution  
>> of
>> this code but that does not solve the DoS issue so only removal of  
>> the
>> code is a real solution.
>
> Removing the function isn't a solution that I'm willing to accept, in
> fact it isn't a solution at all, and since it's so simple you can  
> easily
> do it yourself and spend the time you save on coming up with a  
> sensible
> suggestion.
>
>> Looking for literal strings in /proc is nothing short of stupid!  I  
>> am
>> sorry for you and your users if you can't see that...
>
> I never said that you point was not well taken.
>
> I said I didn't like your tone, given that you were asking for  
> something
> and that continues to be the case. If you think that posting using  
> this
> tone will get priority for fixing this then you are sadly mistaken.

I don't care if you fix it as we are stuck with opensuse 10.3 now for  
the next year and I doubt very much that sort of change would filter  
through...  And we have a workaround by running the automounter using:

perl -e 'exec {"automount"} @ARGV' pwf-amnt ...

Which means that in /proc/*/cmdline the string that appears is "pwf- 
amnt" (as we override argv[0] in the exec call to that effect) thus  
the is_automount_running() function does not find the literal string  
"automount" in there and we can run it as many times as we want (which  
is once per user).

However I think you should fix it because it is a DoS on all servers  
using this daemon for automounting anything important...

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/

Re: Horrible denial of service bug in autmount 5

[Ian Kent]

On Fri, 2008-06-20 at 16:24 +0100, Anton Altaparmakov wrote:
> Which means that in /proc/*/cmdline the string that appears is "pwf- 
> amnt" (as we override argv[0] in the exec call to that effect) thus  
> the is_automount_running() function does not find the literal string  
> "automount" in there and we can run it as many times as we want (which  
> is once per user).

Why do you need to run one instance per user?
What does it get you that using a single source common global map
doesn't provide?

Ian

Re: Horrible denial of service bug in autmount 5

[J.P. King]

Anton has just left for the weekend, so since I can answer this I will

> Why do you need to run one instance per user?
> What does it get you that using a single source common global map
> doesn't provide?

Our maps are generated on the fly by extracting information via LDAP from 
a Novel E directory and then a new automounter is started to mount their 
home directory and a number of other filesystems that we want them to have 
access to.

Because of how NCPFS works we need a separate set of mounts per user.

If we have a shared global map then, since we need to modify it each time 
a user logs in then we have a horrible locking problem.

Oh, and we have approximately 30,000 users, mostly not-entirely-trusted 
students.

Even ignoring that, it is a suboptimal situation that any user can stop 
the automounter from restarting by running a process with a particular 
name.  In general the reason daemons don't try and stop you running 
multiple copies is because it is hard to do even remotely approaching 
right.  Normally they rely on a shared resource acting as your lock - such 
as binding to port 80, or, if they must a lock/pid file.

To address a different question that was asked - why don't we maintain our 
own patched version - clearly we could, however wherever possible we like 
to have bugs, and I _think_ you've accepted that the DOS is a bug, fixed 
upstream so that in the future we won't have to patch it again.

For every item of software that we have to patch we have to track the real 
version for security holes and port our patch to this secured version. 
This is a lot of effort at the best of times, and we have to do it enough 
just to work around the infelicitudes with having to deal with Netware 
without adding to them.

Let me try and turn the problem around.  Given that you accept that the 
DOS is something that you'd like to remove, how would you like to see the 
problem being solved?  Off the top of my head I can think of:

1 command line option - only have it check if the option is set.  This has
   to be the simplest solution, but it isn't great.

2 Pid file - have it check if there is a pid in the pid file before
   writing its own pid.

3 Extend it to check not just the name, but the file/inode of the
   programme being executed.  This has a drawback if you upgrade the binary
   then you can start the new version with the old one still running.  It
   also wouldn't solve our particular problem.

I'm prepared to have the code for 1 or 2 written and a patch submitted for 
you.  I wouldn't put work into 3 since it doesn't help us.  If you can 
come up with a different solution I'm probably happy to have that done too, 
and a patch submitted, so long as it addresses our issue.

> Ian

Julian
--
Julian King
Computer Officer, University of Cambridge, Unix Support

Re: Horrible denial of service bug in autmount 5

[Ian Kent]

On Fri, 2008-06-20 at 16:49 +0100, J.P. King wrote:
> Anton has just left for the weekend, so since I can answer this I will

Oh .. some people have it good.

I must say I wasn't exactly impressed receiving Antons mail after more
than 11 hours of work, but you get that.

> 
> > Why do you need to run one instance per user?
> > What does it get you that using a single source common global map
> > doesn't provide?
> 
> Our maps are generated on the fly by extracting information via LDAP from 
> a Novel E directory and then a new automounter is started to mount their 
> home directory and a number of other filesystems that we want them to have 
> access to.
> 
> Because of how NCPFS works we need a separate set of mounts per user.

Interesting, I haven't heard of a site doing it this way before.

> 
> If we have a shared global map then, since we need to modify it each time 
> a user logs in then we have a horrible locking problem.
> 
> Oh, and we have approximately 30,000 users, mostly not-entirely-trusted 
> students.
> 
> Even ignoring that, it is a suboptimal situation that any user can stop 
> the automounter from restarting by running a process with a particular 
> name.  In general the reason daemons don't try and stop you running 
> multiple copies is because it is hard to do even remotely approaching 
> right.  Normally they rely on a shared resource acting as your lock - such 
> as binding to port 80, or, if they must a lock/pid file.
> 
> To address a different question that was asked - why don't we maintain our 
> own patched version - clearly we could, however wherever possible we like 
> to have bugs, and I _think_ you've accepted that the DOS is a bug, fixed 
> upstream so that in the future we won't have to patch it again.
> 
> For every item of software that we have to patch we have to track the real 
> version for security holes and port our patch to this secured version. 
> This is a lot of effort at the best of times, and we have to do it enough 
> just to work around the infelicitudes with having to deal with Netware 
> without adding to them.

Agreed and understood.

> 
> Let me try and turn the problem around.  Given that you accept that the 
> DOS is something that you'd like to remove, how would you like to see the 
> problem being solved?  Off the top of my head I can think of:
> 
> 1 command line option - only have it check if the option is set.  This has
>    to be the simplest solution, but it isn't great.
> 
> 2 Pid file - have it check if there is a pid in the pid file before
>    writing its own pid.
> 
> 3 Extend it to check not just the name, but the file/inode of the
>    programme being executed.  This has a drawback if you upgrade the binary
>    then you can start the new version with the old one still running.  It
>    also wouldn't solve our particular problem.
> 
> I'm prepared to have the code for 1 or 2 written and a patch submitted for 
> you.  I wouldn't put work into 3 since it doesn't help us.  If you can 
> come up with a different solution I'm probably happy to have that done too, 
> and a patch submitted, so long as it addresses our issue.

No need to have anything written, as Anton says it is fairly straight
forward once it is decided how to go about it.

I just have a few urgent things that need to be done before I can work
on it.

As far as a solution goes, I want to keep the check, in some form, which
I'm not sure of yet, but adding an option to bypass the check is an
obvious and simple way to address the issue for yourselves and I'll do
that to start with.

Ian

Re: Horrible denial of service bug in autmount 5

[J.P. King]

> Oh .. some people have it good.

He did get in rather early, and it has been a rather hard month! 
Yesterday was our deadline for the desktop image (800ish machines) and 
next week we do the remote access servers.

> I must say I wasn't exactly impressed receiving Antons mail after more
> than 11 hours of work, but you get that.

My commiserations, but Anton wasn't to know how your work day had gone.

> As far as a solution goes, I want to keep the check, in some form, which
> I'm not sure of yet, but adding an option to bypass the check is an
> obvious and simple way to address the issue for yourselves and I'll do
> that to start with.

Once we know your solution we'll port that into our setup with the 
expectation that it will have made it into our upstream distribution by 
the time we next upgrade in 9-12 months time.

> Ian

Julian
--
Julian King
Computer Officer, University of Cambridge, Unix Support

Re: Horrible denial of service bug in autmount 5

[Ian Kent]

On Fri, 2008-06-20 at 17:22 +0100, J.P. King wrote:
> > Oh .. some people have it good.
> 
> He did get in rather early, and it has been a rather hard month! 
> Yesterday was our deadline for the desktop image (800ish machines) and 
> next week we do the remote access servers.
> 
> > I must say I wasn't exactly impressed receiving Antons mail after more
> > than 11 hours of work, but you get that.
> 
> My commiserations, but Anton wasn't to know how your work day had gone.
> 
> > As far as a solution goes, I want to keep the check, in some form, which
> > I'm not sure of yet, but adding an option to bypass the check is an
> > obvious and simple way to address the issue for yourselves and I'll do
> > that to start with.
> 
> Once we know your solution we'll port that into our setup with the 
> expectation that it will have made it into our upstream distribution by 
> the time we next upgrade in 9-12 months time.

I'll let you know when I have done it.

Ian

Re: Horrible denial of service bug in autmount 5

[H. Peter Anvin]

Ian Kent wrote:
> On Fri, 2008-06-20 at 16:24 +0100, Anton Altaparmakov wrote:
>> Which means that in /proc/*/cmdline the string that appears is "pwf- 
>> amnt" (as we override argv[0] in the exec call to that effect) thus  
>> the is_automount_running() function does not find the literal string  
>> "automount" in there and we can run it as many times as we want (which  
>> is once per user).
> 
> Why do you need to run one instance per user?
> What does it get you that using a single source common global map
> doesn't provide?

Okay, I'm confused... what reason could there *possibly* be for 
searching /proc/*/cmdline? If there is a need for a mutex of some sort, 
one should typically create a /var/run directory and put in lock files, 
or some other solution to test the mutexing explicitly.  grepping ps, in 
effect, is hardly a good idea, to put it gently.

	-hpa

Re: Horrible denial of service bug in autmount 5

[Ian Kent]

On Fri, 2008-06-20 at 10:03 -0700, H. Peter Anvin wrote:
> Ian Kent wrote:
> > On Fri, 2008-06-20 at 16:24 +0100, Anton Altaparmakov wrote:
> >> Which means that in /proc/*/cmdline the string that appears is "pwf- 
> >> amnt" (as we override argv[0] in the exec call to that effect) thus  
> >> the is_automount_running() function does not find the literal string  
> >> "automount" in there and we can run it as many times as we want (which  
> >> is once per user).
> > 
> > Why do you need to run one instance per user?
> > What does it get you that using a single source common global map
> > doesn't provide?
> 
> Okay, I'm confused... what reason could there *possibly* be for 
> searching /proc/*/cmdline? If there is a need for a mutex of some sort, 
> one should typically create a /var/run directory and put in lock files, 
> or some other solution to test the mutexing explicitly.  grepping ps, in 
> effect, is hardly a good idea, to put it gently.

This is nothing more than a check to see if another instance of
automount(8) is running.

Ian

Re: Horrible denial of service bug in autmount 5

[H. Peter Anvin]

Ian Kent wrote:
>> Okay, I'm confused... what reason could there *possibly* be for 
>> searching /proc/*/cmdline? If there is a need for a mutex of some sort, 
>> one should typically create a /var/run directory and put in lock files, 
>> or some other solution to test the mutexing explicitly.  grepping ps, in 
>> effect, is hardly a good idea, to put it gently.
> 
> This is nothing more than a check to see if another instance of
> automount(8) is running.

But it is an utterly daft way to implement something like that.  If you 
want a lock, create an explicit lock, but doing string-matching on 
command lines is idiotic.

	-hpa

Re: Horrible denial of service bug in autmount 5

[Ian Kent]

On Fri, 2008-06-20 at 12:03 -0700, H. Peter Anvin wrote:
> Ian Kent wrote:
> >> Okay, I'm confused... what reason could there *possibly* be for 
> >> searching /proc/*/cmdline? If there is a need for a mutex of some sort, 
> >> one should typically create a /var/run directory and put in lock files, 
> >> or some other solution to test the mutexing explicitly.  grepping ps, in 
> >> effect, is hardly a good idea, to put it gently.
> > 
> > This is nothing more than a check to see if another instance of
> > automount(8) is running.
> 
> But it is an utterly daft way to implement something like that.  If you 
> want a lock, create an explicit lock, but doing string-matching on 
> command lines is idiotic.

This isn't a lock or anything remotely like it.

And, yes, if run together closely enough the check could easily fail to
work but that isn't what the check is about.

Put another way, running multiple instances of the autofs version 5
daemon isn't supported at the moment.

For the common case usage multiple instances of the daemon aren't
needed.

Re: Horrible denial of service bug in autmount 5

[H. Peter Anvin]

Ian Kent wrote:
>> But it is an utterly daft way to implement something like that.  If you 
>> want a lock, create an explicit lock, but doing string-matching on 
>> command lines is idiotic.
> 
> This isn't a lock or anything remotely like it.

It is a lock, or at least something remotely like it.

> And, yes, if run together closely enough the check could easily fail to
> work but that isn't what the check is about.

What is it about, then?

> Put another way, running multiple instances of the autofs version 5
> daemon isn't supported at the moment.
> 
> For the common case usage multiple instances of the daemon aren't
> needed.

Pardon my earlier abrasiveness (I'm having a horrible day), but why not 
simply have /var/lock/automount and flock() it?

That way there is a sane workaround if the special cases, too.

	-hpa

Re: Horrible denial of service bug in autmount 5

[Ian Kent]

On Fri, 2008-06-20 at 21:27 -0700, H. Peter Anvin wrote:
> Ian Kent wrote:
> >> But it is an utterly daft way to implement something like that.  If you 
> >> want a lock, create an explicit lock, but doing string-matching on 
> >> command lines is idiotic.
> > 
> > This isn't a lock or anything remotely like it.
> 
> It is a lock, or at least something remotely like it.
> 
> > And, yes, if run together closely enough the check could easily fail to
> > work but that isn't what the check is about.
> 
> What is it about, then?
> 
> > Put another way, running multiple instances of the autofs version 5
> > daemon isn't supported at the moment.
> > 
> > For the common case usage multiple instances of the daemon aren't
> > needed.
> 
> Pardon my earlier abrasiveness (I'm having a horrible day), but why not 
> simply have /var/lock/automount and flock() it?

True and I very nearly wrote back and said that but then I started to
remember the painful issues of SIGKILL and trying to work out if the
flag file is really valid and the issues of mount(8) using a lock file
with flock() and stopped myself.

But, yes, I could do that.

To support multiple instances I will need to check in a few places that
I don't interfere with other process mounts. I suspect the Cambridge
guys have either been lucky so far and possibly not properly
investigated the issue. They don't seem to want to discuss it either so
I fear their in for an unpleasant surprise at some point.

Ian

Re: Horrible denial of service bug in autmount 5

[H. Peter Anvin]

Ian Kent wrote:
> 
> True and I very nearly wrote back and said that but then I started to
> remember the painful issues of SIGKILL and trying to work out if the
> flag file is really valid and the issues of mount(8) using a lock file
> with flock() and stopped myself.
> 

I think the issues with mount(8) had to do with backwards compatibility; 
after all, it's one of the very first Linux utilities at all.

> But, yes, I could do that.
> 
> To support multiple instances I will need to check in a few places that
> I don't interfere with other process mounts. I suspect the Cambridge
> guys have either been lucky so far and possibly not properly
> investigated the issue. They don't seem to want to discuss it either so
> I fear their in for an unpleasant surprise at some point.

Yeah, that's another ball of wax entirely.

	-hpa

Re: Horrible denial of service bug in autmount 5

[J.P. King]

> To support multiple instances I will need to check in a few places that
> I don't interfere with other process mounts. I suspect the Cambridge
> guys have either been lucky so far and possibly not properly
> investigated the issue.

The Cambridge guys have been using earlier versions of autofs which didn't 
have this restriction (currently 4.1).  Whether there was a bug that we 
were sometimes triggering is a different matter.  I am not aware of the 
potential issue that you think we could be hitting, and I am unclear how 
it could be something that was automount's problem, as opposed to a 
general mounting filesystems race condition.

> They don't seem to want to discuss it either so
> I fear their in for an unpleasant surprise at some point.

Not only do we want to discuss it, we have been.  We are also happy to put 
work into resolving the issue.  However we have identified a trivial DOS
and some, frankly, poor code, and there seems to be some reluctance in 
accepting that this is an issue, regardless of any specific problems we're 
having.

> Ian

Julian
--
Julian King
Computer Officer, University of Cambridge, Unix Support

Re: Horrible denial of service bug in autmount 5

[Ian Kent]

On Sat, 2008-06-21 at 10:18 +0100, J.P. King wrote:
> > To support multiple instances I will need to check in a few places that
> > I don't interfere with other process mounts. I suspect the Cambridge
> > guys have either been lucky so far and possibly not properly
> > investigated the issue.
> 
> The Cambridge guys have been using earlier versions of autofs which didn't 
> have this restriction (currently 4.1).  Whether there was a bug that we 
> were sometimes triggering is a different matter.  I am not aware of the 
> potential issue that you think we could be hitting, and I am unclear how 
> it could be something that was automount's problem, as opposed to a 
> general mounting filesystems race condition.

Sure, but version 5 is very different from version 4 and because it is
meant to run as a single instance there is potential for conflict.

It may well be the case that indirect maps are fine as they generally
restrict themselves to within their mount point but I'm fairly sure that
direct maps won't be OK.

> 
> > They don't seem to want to discuss it either so
> > I fear their in for an unpleasant surprise at some point.
> 
> Not only do we want to discuss it, we have been.  We are also happy to put 
> work into resolving the issue.  However we have identified a trivial DOS
> and some, frankly, poor code, and there seems to be some reluctance in 
> accepting that this is an issue, regardless of any specific problems we're 
> having.

No, I'm not reluctant to accept that it is an issue but you need to
consider that running multiple instances may not be the right way to
achieve what you need with this version of autofs.

Is there some reason you can't update the master map and send a HUP
signal to the daemon?

Ian

Re: Horrible denial of service bug in autmount 5

[J.P. King]

> Is there some reason you can't update the master map and send a HUP
> signal to the daemon?

As I said previously, because this is quite hard.  The maps need to be 
changed by the login process.  We currently do this in the PAM 
configuration.  Given that we can have multiple log in at the same time 
this means we need to do a lot of careful locking.

Ideally we would do something like include all the files from a 
configuration directory and have a HUP get it to re-read all these 
configuration snippets.  This would be a relatively involved change 
however.

Please do remember where we came in though - we have our needs, which is 
our problem.  You have a bug/DOS which we identified whilst looking for a 
reason that something had broken between OpenSUSE 10.2 and OpenSUSE 10.3.
Clearly we are going to hope that when you fix your bug we can backport 
the solution and solve our problem.  If we can't we can't, and we will 
work around whatever issues we have left.

> Ian

Julian
--
Julian King
Computer Officer, University of Cambridge, Unix Support

Re: Horrible denial of service bug in autmount 5

[H. Peter Anvin]

J.P. King wrote:
> 
> As I said previously, because this is quite hard.  The maps need to be 
> changed by the login process.  We currently do this in the PAM 
> configuration.  Given that we can have multiple log in at the same time 
> this means we need to do a lot of careful locking.
> 
> Ideally we would do something like include all the files from a 
> configuration directory and have a HUP get it to re-read all these 
> configuration snippets.  This would be a relatively involved change 
> however.
> 

This sounds like a perfect use of an executable map to me.  I don't know 
exactly what your maps look like, but it seems to me that since you have 
an unusual local requirement it would make sense to deal with it in a 
manner external to autofs itself.

	-hpa

Re: Horrible denial of service bug in autmount 5

[Jim Carter]

On Sun, 22 Jun 2008, H. Peter Anvin wrote:

> J.P. King wrote:
> > Ideally we would do something like include all the files from a 
> > configuration directory and have a HUP get it to re-read all these 
> > configuration snippets.  This would be a relatively involved change 
> > however.
> > 
> 
> This sounds like a perfect use of an executable map to me.  I don't know 
> exactly what your maps look like, but it seems to me that since you have 
> an unusual local requirement it would make sense to deal with it in a 
> manner external to autofs itself.

That's an excellent suggestion!  At my site we used an executable map to 
get the effect of submounts, before I figured out how to do the job with 
submounts.  It was reliable and reasonably efficient.

James F. Carter          Voice 310 825 2897    FAX 310 206 6673
UCLA-Mathnet;  6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA 90095-1555
Email: jimc@math.ucla.edu  http://www.math.ucla.edu/~jimc (q.v. for PGP key)

Re: Horrible denial of service bug in autmount 5

[Anton Altaparmakov]

Hi,

On 23 Jun 2008, at 02:08, H. Peter Anvin wrote:
> J.P. King wrote:
>> As I said previously, because this is quite hard.  The maps need to  
>> be changed by the login process.  We currently do this in the PAM  
>> configuration.  Given that we can have multiple log in at the same  
>> time this means we need to do a lot of careful locking.
>> Ideally we would do something like include all the files from a  
>> configuration directory and have a HUP get it to re-read all these  
>> configuration snippets.  This would be a relatively involved change  
>> however.
>
> This sounds like a perfect use of an executable map to me.

Sounds like something that is certainly worth investigating.

> I don't know exactly what your maps look like, but it seems to me  
> that since you have an unusual local requirement it would make sense  
> to deal with it in a manner external to autofs itself.


This is what ours look like at present.  When a user logs in we  
generate two files (below $user is the user name of the user logging  
in):

/var/run/pwfautomount/master.$user
/var/run/pwfautomount/auto.$user

I.e. for me with user name aia21 the files are:

/var/run/pwfautomount/master.aia21
/var/run/pwfautomount/auto.aia21

Where the first file is the master map for that user and it contains a  
single line like this (in this example the user logging in is me, i.e.  
aia21, hence the two "aia21" strings, those would be set to the  
currently logging in user):

/servers/aia21 file:/var/run/pwfautomount/auto.aia21

And the second file contains at present for my user "aia21":

PWF-HOME1	- 
fstype 
= 
ncp 
,ipserver 
= 
172.16.3.42 
,owner 
= 
aia21 
,uid 
= 
aia21 
,gid 
= 
aia21 
,hard 
,nodev 
,nosuid 
,nfsextras 
,strong,codepage=cp850,iocharset=utf8,filemode=640,dirmode=750,a=/ 
authcon/aia21	:PWF-HOME1/aia21.Aliases
PWF-HOME2	- 
fstype 
= 
ncp 
,ipserver 
= 
172.16.3.43 
,owner 
= 
aia21 
,uid 
= 
aia21 
,gid 
= 
aia21 
,hard 
,nodev 
,nosuid 
,nfsextras 
,strong,codepage=cp850,iocharset=utf8,filemode=640,dirmode=750,a=/ 
authcon/aia21	:PWF-HOME2/aia21.Aliases
PWF-HOME3	- 
fstype 
= 
ncp 
,ipserver 
= 
172.16.3.44 
,owner 
= 
aia21 
,uid 
= 
aia21 
,gid 
= 
aia21 
,hard 
,nodev 
,nosuid 
,nfsextras 
,strong,codepage=cp850,iocharset=utf8,filemode=640,dirmode=750,a=/ 
authcon/aia21	:PWF-HOME3/aia21.Aliases
PWF-HOME4	- 
fstype 
= 
ncp 
,ipserver 
= 
172.16.3.45 
,owner 
= 
aia21 
,uid 
= 
aia21 
,gid 
= 
aia21 
,hard 
,nodev 
,nosuid 
,nfsextras 
,strong,codepage=cp850,iocharset=utf8,filemode=640,dirmode=750,a=/ 
authcon/aia21	:PWF-HOME4/aia21.Aliases
PWF-SHARED	- 
fstype 
= 
ncp 
,ipserver 
= 
172.16.3.8 
,owner 
= 
aia21 
,uid 
= 
aia21 
,gid 
= 
aia21 
,hard 
,nodev 
,nosuid 
,nfsextras 
,strong,codepage=cp850,iocharset=utf8,filemode=640,dirmode=750,a=/ 
authcon/aia21	:PWF-SHARED/aia21.Aliases

Note that these lines are generated by querying the Netware e- 
Directory and have to be regenerated each time a user logs in as the  
list can have changed between logins and in particular the list can be  
different between different people logging in.

We then launch the automount process like so:

perl -e 'exec {"automount"} @ARGV' pwf-amnt \
     -p "/var/run/pwfautomount/pid.$user" "/var/run/pwfautomount/ 
master.$user"

Note the perl exec magic is only needed with the new automount from  
autofs 5.  With autofs 4 we simply ran automount with appropriate  
options...


Further, when a user logs out, we ask the automount process for that  
user to shut down, thus causing all those mounts to disappear like so:

# Tidy up the user's automounter(s)
ampid=$(cat "/var/run/pwfautomount/pid.$user")

# Kill user's processes politely at first, then less so.
su -c "kill -TERM -1" $user
sleep 1
su -c "kill -KILL -1" $user
sleep 1

rm -f /var/run/pwfautomount/auto.$user /var/run/pwfautomount/pw.$user \
       /var/run/pwfautomount/master.$user

# Repeatedly ask the automounter to shut down until it succeeds (and  
hence the
# "kill" fails).
(
     while kill -USR2 $ampid 2>/dev/null; do
         sleep 1
     done
) >/dev/null 2>&1

So we have a direct benefit of having multiple automount instances  
because it makes it very easy to cause all the user's automounts to be  
unmounted on logout.

We do not want them to remain as the number of connections to the  
Novell e-Directory is limited per user thus if the automount mounts  
were to remain for some reason each of them would count as a  
"connection" and a user could easily get into a situation where they  
can't log in at all any more because of old automount mounts sitting  
around on various other servers/workstations.

How would we be able to perform such cleanup if using one automount  
instance with an executable map?

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/

Re: Horrible denial of service bug in autmount 5

[Ian Kent]

On Mon, 2008-06-23 at 09:06 +0100, Anton Altaparmakov wrote:
> Hi,
> 
> On 23 Jun 2008, at 02:08, H. Peter Anvin wrote:
> > J.P. King wrote:
> >> As I said previously, because this is quite hard.  The maps need to  
> >> be changed by the login process.  We currently do this in the PAM  
> >> configuration.  Given that we can have multiple log in at the same  
> >> time this means we need to do a lot of careful locking.
> >> Ideally we would do something like include all the files from a  
> >> configuration directory and have a HUP get it to re-read all these  
> >> configuration snippets.  This would be a relatively involved change  
> >> however.
> >
> > This sounds like a perfect use of an executable map to me.
> 
> Sounds like something that is certainly worth investigating.

I suspect that you'll find that some existing bugs will prevent you from
moving to a single master map with 5.0.2 (and 5.0.3 as I'm currently
working on problems submount handling). I also think there is more than
one way to do this as well but there is no way to do a forced expire on
a specific mount point at the moment.

I'll just walk through the email and try to describe an alternate
approach in spite of the caveats above. I'm not sure that my suggestions
will be quite what you need, specifically wrt. the need for a forced
expire on a specific automount.

> 
> > I don't know exactly what your maps look like, but it seems to me  
> > that since you have an unusual local requirement it would make sense  
> > to deal with it in a manner external to autofs itself.
> 
> 
> This is what ours look like at present.  When a user logs in we  
> generate two files (below $user is the user name of the user logging  
> in):
> 
> /var/run/pwfautomount/master.$user
> /var/run/pwfautomount/auto.$user
> 
> I.e. for me with user name aia21 the files are:
> 
> /var/run/pwfautomount/master.aia21
> /var/run/pwfautomount/auto.aia21
> 
> Where the first file is the master map for that user and it contains a  
> single line like this (in this example the user logging in is me, i.e.  
> aia21, hence the two "aia21" strings, those would be set to the  
> currently logging in user):
> 
> /servers/aia21 file:/var/run/pwfautomount/auto.aia21

Here we might use:
/servers	file:/<any dir you want>/auto.user

For the map entries below this might be something like:
* -fstype=autofs file:/var/run/pwfautomount/auto.&

The point of using a submount is that autofs will mount /servers/<user>
(for issuing control commands) and get it's map entries from the
map /var/run/pwfautomount/auto.<user> and when it expires away the mount
itself will also go away. Quite similar to what you have now and should
actually work with both v4 and v5 (or would if I didn't have bugs with
submounts in v5 at the moment and if v4 had a forced expire).

The problem for your setup is be not being able to send a forced expire
(USR2) to a specific mount which will of course cause mounts you don't
want umounted to be umounted. This is an example of the type of control
function we want to provide either by using the daemon or another
control program. The way this would work is similar to the way in which
we dynamically change the log level now (sorry, added in 5.0.3),
optionally specifying an autofs mount point.

But, yes, it ain't there now.

> 
> And the second file contains at present for my user "aia21":
> 
> PWF-HOME1	- 
> fstype 
> = 
> ncp 
> ,ipserver 
> = 
> 172.16.3.42 
> ,owner 
> = 
> aia21 
> ,uid 
> = 
> aia21 
> ,gid 
> = 
> aia21 
> ,hard 
> ,nodev 
> ,nosuid 
> ,nfsextras 
> ,strong,codepage=cp850,iocharset=utf8,filemode=640,dirmode=750,a=/ 
> authcon/aia21	:PWF-HOME1/aia21.Aliases
> PWF-HOME2	- 
> fstype 
> = 
> ncp 
> ,ipserver 
> = 
> 172.16.3.43 
> ,owner 
> = 
> aia21 
> ,uid 
> = 
> aia21 
> ,gid 
> = 
> aia21 
> ,hard 
> ,nodev 
> ,nosuid 
> ,nfsextras 
> ,strong,codepage=cp850,iocharset=utf8,filemode=640,dirmode=750,a=/ 
> authcon/aia21	:PWF-HOME2/aia21.Aliases
> PWF-HOME3	- 
> fstype 
> = 
> ncp 
> ,ipserver 
> = 
> 172.16.3.44 
> ,owner 
> = 
> aia21 
> ,uid 
> = 
> aia21 
> ,gid 
> = 
> aia21 
> ,hard 
> ,nodev 
> ,nosuid 
> ,nfsextras 
> ,strong,codepage=cp850,iocharset=utf8,filemode=640,dirmode=750,a=/ 
> authcon/aia21	:PWF-HOME3/aia21.Aliases
> PWF-HOME4	- 
> fstype 
> = 
> ncp 
> ,ipserver 
> = 
> 172.16.3.45 
> ,owner 
> = 
> aia21 
> ,uid 
> = 
> aia21 
> ,gid 
> = 
> aia21 
> ,hard 
> ,nodev 
> ,nosuid 
> ,nfsextras 
> ,strong,codepage=cp850,iocharset=utf8,filemode=640,dirmode=750,a=/ 
> authcon/aia21	:PWF-HOME4/aia21.Aliases
> PWF-SHARED	- 
> fstype 
> = 
> ncp 
> ,ipserver 
> = 
> 172.16.3.8 
> ,owner 
> = 
> aia21 
> ,uid 
> = 
> aia21 
> ,gid 
> = 
> aia21 
> ,hard 
> ,nodev 
> ,nosuid 
> ,nfsextras 
> ,strong,codepage=cp850,iocharset=utf8,filemode=640,dirmode=750,a=/ 
> authcon/aia21	:PWF-SHARED/aia21.Aliases
> 
> Note that these lines are generated by querying the Netware e- 
> Directory and have to be regenerated each time a user logs in as the  
> list can have changed between logins and in particular the list can be  
> different between different people logging in.
> 
> We then launch the automount process like so:
> 
> perl -e 'exec {"automount"} @ARGV' pwf-amnt \
>      -p "/var/run/pwfautomount/pid.$user" "/var/run/pwfautomount/ 
> master.$user"
> 
> Note the perl exec magic is only needed with the new automount from  
> autofs 5.  With autofs 4 we simply ran automount with appropriate  
> options...
> 
> 
> Further, when a user logs out, we ask the automount process for that  
> user to shut down, thus causing all those mounts to disappear like so:
> 
> # Tidy up the user's automounter(s)
> ampid=$(cat "/var/run/pwfautomount/pid.$user")
> 
> # Kill user's processes politely at first, then less so.
> su -c "kill -TERM -1" $user
> sleep 1
> su -c "kill -KILL -1" $user
> sleep 1
> 
> rm -f /var/run/pwfautomount/auto.$user /var/run/pwfautomount/pw.$user \
>        /var/run/pwfautomount/master.$user
> 
> # Repeatedly ask the automounter to shut down until it succeeds (and  
> hence the
> # "kill" fails).
> (
>      while kill -USR2 $ampid 2>/dev/null; do
>          sleep 1
>      done
> ) >/dev/null 2>&1

We would probably take this logic into the forced-expire control
function, but for the general case, where forced umounting might not be
enabled, there's also the question of how long we try to get rid of the
mounts and the form of the communication of the result to the requester
(possibly a numeric return code?).

> 
> So we have a direct benefit of having multiple automount instances  
> because it makes it very easy to cause all the user's automounts to be  
> unmounted on logout.

And probably will need continue this way for a while yet I think due to
the USR2 issue.

> 
> We do not want them to remain as the number of connections to the  
> Novell e-Directory is limited per user thus if the automount mounts  
> were to remain for some reason each of them would count as a  
> "connection" and a user could easily get into a situation where they  
> can't log in at all any more because of old automount mounts sitting  
> around on various other servers/workstations.
> 
> How would we be able to perform such cleanup if using one automount  
> instance with an executable map?

Perhaps someone else could post an approach using an executable map but 

I'm not sure we can get around the forced-expire difficulty though.

Ian

Re: Horrible denial of service bug in autmount 5

[Ian Kent]

On Mon, 2008-06-23 at 17:03 +0800, Ian Kent wrote:
> > 
> > /servers/aia21 file:/var/run/pwfautomount/auto.aia21
> 
> Here we might use:
> /servers	file:/<any dir you want>/auto.user
> 
> For the map entries below this might be something like:
> * -fstype=autofs file:/var/run/pwfautomount/auto.&
> 

Clearly this could well be "program:/var/run/pwfautomount/auto.&" and
contain the logic to generate the mount location. But it would be called
to lookup the mount location whenever a request to mount a given <key>
comes in (eg. /devsers/<user>/<key>) which may introduce unacceptable
overhead.

If this was used then the program map would return:

-fstype=autofs file:/var/run/pwfautomount/auto.&

without the key itself.

Ian

Re: Horrible denial of service bug in autmount 5

[Ian Kent]

On Mon, 2008-06-23 at 17:12 +0800, Ian Kent wrote:
> On Mon, 2008-06-23 at 17:03 +0800, Ian Kent wrote:
> > > 
> > > /servers/aia21 file:/var/run/pwfautomount/auto.aia21
> > 
> > Here we might use:
> > /servers	file:/<any dir you want>/auto.user
> > 
> > For the map entries below this might be something like:
> > * -fstype=autofs file:/var/run/pwfautomount/auto.&
> > 
> 
> Clearly this could well be "program:/var/run/pwfautomount/auto.&" and
> contain the logic to generate the mount location. But it would be called
> to lookup the mount location whenever a request to mount a given <key>
> comes in (eg. /devsers/<user>/<key>) which may introduce unacceptable
> overhead.
> 
> If this was used then the program map would return:
> 
> -fstype=autofs file:/var/run/pwfautomount/auto.&

Oops, that's not right.
It would return the location corresponding to the passed in key.

Ignore my comment about a program map, it's not right, I'll need to
re-think that.

Ian

Re: Horrible denial of service bug in autmount 5

[Jim Carter]

On Mon, 23 Jun 2008, Anton Altaparmakov wrote:

> On 23 Jun 2008, at 02:08, H. Peter Anvin wrote:
> > This sounds like a perfect use of an executable map to me.
> 
> Sounds like something that is certainly worth investigating.
> ......
> This is what ours look like at present.  When a user logs in we  
> generate two files (below $user is the user name of the user logging  
> in):

This problem is interesting: how would you do this while using the 
multi-thread capability of autofs version 5?  Here's how far I got.  First, 
we're not going to mount anything at login time; we're going to mount the 
homedir the first time it's referenced, which on my system would be 
slightly before logging in is finished (to check the ssh key in the 
homedir).  Similarly, we're going to rely on automount's normal expiration 
behavior to get rid of mounts after logout.

/etc/auto.master would say:
/server		/etc/auto.per-user

/etc/auto.per-user says:
*       -fstype=autofs,-DNCP_USER=&     program:/etc/auto.userdirs
    (this is bogus, see below)

/etc/auto.userdirs is executable and goes something like this:  (I modified 
it so it reports something that can be mounted on my system, and it 
actually happens.)

#!/bin/sh
# $1 is the map key, i.e. on access to /server/$user/$share, $1 would be 
# $share.  Translate the key into the server's IP address.  Likely this
# is waaaaay different on an authentic Novell network.
ip=`dig +short +search $1 A`

# Emit the map row (leaving off the key).  Bogus \'s supplied for 
# readability.
echo -fstype=ncp,ipserver=$ip,owner=$NCP_USER,uid=$NCP_USER,gid=$NCP_USER,\
hard,nodev,nosuid,nfsextras,strong,codepage=cp850,iocharset=utf8,\
filemode=640,dirmode=750,a=/authcon/$NCP_USER    :$1/$NCP_USER.Aliases

Now, what's the right way to get the NCP user available to the program 
map?  In a file map -DNCP_USER (as shown, with the hyphen) will propagate 
this variable into the map rows, so the whole thing could have been
handled in a file map...  except for the IP address in the mount options. 
If ipserver could possibly be a FQDN rather than a numeric IP address, I 
would be very, very tempted to create a massive DNS map that has "A" 
records for PWF-HOME$N.$user.ncp-xlate.cam.ac.uk (to give an example domain 
structure).  This would be updated, perhaps by dynamic DNS updates, 
whenever a user's server assignment changed.  

Returning to the question that most interests me, in the submount I tried 
these variants:

*       -fstype=autofs,-DNCP_USER=&     program:/etc/auto.userdirs
    (NCP_USER was not defined as an environment variable; neither was USER)
*       -fstype=autofs     program:/etc/auto.userdirs &
    (The extra arg to the program map was ignored with no error message)

What do people think would be the best way to get the containing map key 
into the program map?  Not the key to be translated by the map (in our 
case PWF-HOME1 etc.), but the key that caused the submount to be created, 
or any of the other environment variables of the submount, such as $USER, 
which is probably what you really want to use here.  

Warning: before deploying a submount-based scheme in production, watch for 
a resolution of the internal locking (mutex) issue that's so bedeviling me.

James F. Carter          Voice 310 825 2897    FAX 310 206 6673
UCLA-Mathnet;  6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA 90095-1555
Email: jimc@math.ucla.edu  http://www.math.ucla.edu/~jimc (q.v. for PGP key)

Re: Horrible denial of service bug in autmount 5

[Anton Altaparmakov]

Hi,

On 24 Jun 2008, at 02:08, Jim Carter wrote:
> On Mon, 23 Jun 2008, Anton Altaparmakov wrote:
>> On 23 Jun 2008, at 02:08, H. Peter Anvin wrote:
>>> This sounds like a perfect use of an executable map to me.
>>
>> Sounds like something that is certainly worth investigating.
>> ......
>> This is what ours look like at present.  When a user logs in we
>> generate two files (below $user is the user name of the user logging
>> in):
>
> This problem is interesting: how would you do this while using the
> multi-thread capability of autofs version 5?  Here's how far I got.   
> First,
> we're not going to mount anything at login time; we're going to  
> mount the
> homedir the first time it's referenced, which on my system would be
> slightly before logging in is finished (to check the ssh key in the
> homedir).

Note this is not possible for us.  You cannot use SSH-keys when the  
home directory is on a Netware server because you need the user's  
password to be able to mount the home directory, there is no way to  
perform the mount before asking the user for their password.  Once the  
user is logged in, as long as they maintain that session open, they  
can then use SSH key based authentication when logging in to the same  
PWF Linux server as their home directory is already mounted so their  
SSH keys are accessible before their subsequent logins.  This is one  
of the "features" our users complain about the most...

> Similarly, we're going to rely on automount's normal expiration
> behavior to get rid of mounts after logout.
>
> /etc/auto.master would say:
> /server		/etc/auto.per-user
>
> /etc/auto.per-user says:
> *       -fstype=autofs,-DNCP_USER=&     program:/etc/auto.userdirs
>    (this is bogus, see below)
>
> /etc/auto.userdirs is executable and goes something like this:  (I  
> modified
> it so it reports something that can be mounted on my system, and it
> actually happens.)
>
> #!/bin/sh
> # $1 is the map key, i.e. on access to /server/$user/$share, $1  
> would be
> # $share.  Translate the key into the server's IP address.  Likely  
> this
> # is waaaaay different on an authentic Novell network.
> ip=`dig +short +search $1 A`
>
> # Emit the map row (leaving off the key).  Bogus \'s supplied for
> # readability.
> echo -fstype=ncp,ipserver=$ip,owner=$NCP_USER,uid=$NCP_USER,gid= 
> $NCP_USER,\
> hard,nodev,nosuid,nfsextras,strong,codepage=cp850,iocharset=utf8,\
> filemode=640,dirmode=750,a=/authcon/$NCP_USER    :$1/$NCP_USER.Aliases
>
> Now, what's the right way to get the NCP user available to the program
> map?  In a file map -DNCP_USER (as shown, with the hyphen) will  
> propagate
> this variable into the map rows, so the whole thing could have been
> handled in a file map...  except for the IP address in the mount  
> options.
> If ipserver could possibly be a FQDN rather than a numeric IP  
> address, I
> would be very, very tempted to create a massive DNS map that has "A"
> records for PWF-HOME$N.$user.ncp-xlate.cam.ac.uk (to give an example  
> domain
> structure).  This would be updated, perhaps by dynamic DNS updates,
> whenever a user's server assignment changed.
>
> Returning to the question that most interests me, in the submount I  
> tried
> these variants:
>
> *       -fstype=autofs,-DNCP_USER=&     program:/etc/auto.userdirs
>    (NCP_USER was not defined as an environment variable; neither was  
> USER)
> *       -fstype=autofs     program:/etc/auto.userdirs &
>    (The extra arg to the program map was ignored with no error  
> message)
>
> What do people think would be the best way to get the containing map  
> key
> into the program map?  Not the key to be translated by the map (in our
> case PWF-HOME1 etc.), but the key that caused the submount to be  
> created,
> or any of the other environment variables of the submount, such as  
> $USER,
> which is probably what you really want to use here.

I would have expected the -DNCP_USER= to be created as an environment  
variable for the program map to find or alternatively to be given on  
the program map command line.  Given a choice the environment variable  
option would be much nicer and easier to use.

> Warning: before deploying a submount-based scheme in production,  
> watch for
> a resolution of the internal locking (mutex) issue that's so  
> bedeviling me.

Could you elaborate on this?  What issue are you seeing?  Thanks a lot  
in advance!

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/

Re: Horrible denial of service bug in autmount 5

[Ian Kent]

On Tue, 2008-06-24 at 08:27 +0100, Anton Altaparmakov wrote:
> Hi,
> 
> On 24 Jun 2008, at 02:08, Jim Carter wrote:
> > On Mon, 23 Jun 2008, Anton Altaparmakov wrote:
> >> On 23 Jun 2008, at 02:08, H. Peter Anvin wrote:
> >>> This sounds like a perfect use of an executable map to me.
> >>
> >> Sounds like something that is certainly worth investigating.
> >> ......
> >> This is what ours look like at present.  When a user logs in we
> >> generate two files (below $user is the user name of the user logging
> >> in):
> >
> > This problem is interesting: how would you do this while using the
> > multi-thread capability of autofs version 5?  Here's how far I got.   
> > First,
> > we're not going to mount anything at login time; we're going to  
> > mount the
> > homedir the first time it's referenced, which on my system would be
> > slightly before logging in is finished (to check the ssh key in the
> > homedir).
> 
> Note this is not possible for us.  You cannot use SSH-keys when the  
> home directory is on a Netware server because you need the user's  
> password to be able to mount the home directory, there is no way to  
> perform the mount before asking the user for their password.  Once the  
> user is logged in, as long as they maintain that session open, they  
> can then use SSH key based authentication when logging in to the same  
> PWF Linux server as their home directory is already mounted so their  
> SSH keys are accessible before their subsequent logins.  This is one  
> of the "features" our users complain about the most...
> 
> > Similarly, we're going to rely on automount's normal expiration
> > behavior to get rid of mounts after logout.
> >
> > /etc/auto.master would say:
> > /server		/etc/auto.per-user
> >
> > /etc/auto.per-user says:
> > *       -fstype=autofs,-DNCP_USER=&     program:/etc/auto.userdirs
> >    (this is bogus, see below)
> >
> > /etc/auto.userdirs is executable and goes something like this:  (I  
> > modified
> > it so it reports something that can be mounted on my system, and it
> > actually happens.)
> >
> > #!/bin/sh
> > # $1 is the map key, i.e. on access to /server/$user/$share, $1  
> > would be
> > # $share.  Translate the key into the server's IP address.  Likely  
> > this
> > # is waaaaay different on an authentic Novell network.
> > ip=`dig +short +search $1 A`
> >
> > # Emit the map row (leaving off the key).  Bogus \'s supplied for
> > # readability.
> > echo -fstype=ncp,ipserver=$ip,owner=$NCP_USER,uid=$NCP_USER,gid= 
> > $NCP_USER,\
> > hard,nodev,nosuid,nfsextras,strong,codepage=cp850,iocharset=utf8,\
> > filemode=640,dirmode=750,a=/authcon/$NCP_USER    :$1/$NCP_USER.Aliases
> >
> > Now, what's the right way to get the NCP user available to the program
> > map?  In a file map -DNCP_USER (as shown, with the hyphen) will  
> > propagate
> > this variable into the map rows, so the whole thing could have been
> > handled in a file map...  except for the IP address in the mount  
> > options.
> > If ipserver could possibly be a FQDN rather than a numeric IP  
> > address, I
> > would be very, very tempted to create a massive DNS map that has "A"
> > records for PWF-HOME$N.$user.ncp-xlate.cam.ac.uk (to give an example  
> > domain
> > structure).  This would be updated, perhaps by dynamic DNS updates,
> > whenever a user's server assignment changed.
> >
> > Returning to the question that most interests me, in the submount I  
> > tried
> > these variants:
> >
> > *       -fstype=autofs,-DNCP_USER=&     program:/etc/auto.userdirs
> >    (NCP_USER was not defined as an environment variable; neither was  
> > USER)
> > *       -fstype=autofs     program:/etc/auto.userdirs &
> >    (The extra arg to the program map was ignored with no error  
> > message)
> >
> > What do people think would be the best way to get the containing map  
> > key
> > into the program map?  Not the key to be translated by the map (in our
> > case PWF-HOME1 etc.), but the key that caused the submount to be  
> > created,
> > or any of the other environment variables of the submount, such as  
> > $USER,
> > which is probably what you really want to use here.
> 
> I would have expected the -DNCP_USER= to be created as an environment  
> variable for the program map to find or alternatively to be given on  
> the program map command line.  Given a choice the environment variable  
> option would be much nicer and easier to use.

Setting them in the environment is the sensible and easiest thing for me
to do. But setting the macros that are in the lookup table at the time
of the mount it isn't done at the moment. I think we had a request for
this some time ago and I made a patch for testing but didn't get any
feedback so I let it slip.

> 
> > Warning: before deploying a submount-based scheme in production,  
> > watch for
> > a resolution of the internal locking (mutex) issue that's so  
> > bedeviling me.
> 
> Could you elaborate on this?  What issue are you seeing?  Thanks a lot  
> in advance!

Jim sees autofs hang.
There is synchronization problem between submounts that are shutting
down at the same time a mount request comes in. Of course we only see it
on busy systems.

Ian

Re: Horrible denial of service bug in autmount 5

[Anton Altaparmakov]

Hi,

On 24 Jun 2008, at 08:38, Ian Kent wrote:
> On Tue, 2008-06-24 at 08:27 +0100, Anton Altaparmakov wrote:
>> On 24 Jun 2008, at 02:08, Jim Carter wrote:
>>> What do people think would be the best way to get the containing map
>>> key
>>> into the program map?  Not the key to be translated by the map (in  
>>> our
>>> case PWF-HOME1 etc.), but the key that caused the submount to be
>>> created,
>>> or any of the other environment variables of the submount, such as
>>> $USER,
>>> which is probably what you really want to use here.
>>
>> I would have expected the -DNCP_USER= to be created as an environment
>> variable for the program map to find or alternatively to be given on
>> the program map command line.  Given a choice the environment  
>> variable
>> option would be much nicer and easier to use.
>
> Setting them in the environment is the sensible and easiest thing  
> for me
> to do. But setting the macros that are in the lookup table at the time
> of the mount it isn't done at the moment. I think we had a request for
> this some time ago and I made a patch for testing but didn't get any
> feedback so I let it slip.

Ok.  I have for testing purposes done the changes you suggested and am  
using a file map successfully so we probably will not want/need to use  
a program map given the overhead it would impose.  Much easier to  
regenerate the file map for each user when they log in.  We now have:

/etc/auto.master:
/.servers	file:/etc/auto.user

/etc/auto.user:
* -fstype=autofs file:/var/run/pwfautomount/auto.&

And /var/run/pwfautomount/auto.<username> is generated when <username>  
user logs in and for my user aia21 contains exactly the same content  
as before:

PWF-HOME1	- 
fstype 
= 
ncp 
,ipserver 
= 
172.16.3.42 
,owner 
= 
aia21 
,uid 
= 
aia21 
,gid 
= 
aia21 
,hard 
,nodev 
,nosuid 
,nfsextras 
,strong,codepage=cp850,iocharset=utf8,filemode=640,dirmode=750,a=/ 
authcon/aia21	:PWF-HOME1/aia21.Aliases
PWF-HOME2	- 
fstype 
= 
ncp 
,ipserver 
= 
172.16.3.43 
,owner 
= 
aia21 
,uid 
= 
aia21 
,gid 
= 
aia21 
,hard 
,nodev 
,nosuid 
,nfsextras 
,strong,codepage=cp850,iocharset=utf8,filemode=640,dirmode=750,a=/ 
authcon/aia21	:PWF-HOME2/aia21.Aliases
PWF-HOME3	- 
fstype 
= 
ncp 
,ipserver 
= 
172.16.3.44 
,owner 
= 
aia21 
,uid 
= 
aia21 
,gid 
= 
aia21 
,hard 
,nodev 
,nosuid 
,nfsextras 
,strong,codepage=cp850,iocharset=utf8,filemode=640,dirmode=750,a=/ 
authcon/aia21	:PWF-HOME3/aia21.Aliases
PWF-HOME4	- 
fstype 
= 
ncp 
,ipserver 
= 
172.16.3.45 
,owner 
= 
aia21 
,uid 
= 
aia21 
,gid 
= 
aia21 
,hard 
,nodev 
,nosuid 
,nfsextras 
,strong,codepage=cp850,iocharset=utf8,filemode=640,dirmode=750,a=/ 
authcon/aia21	:PWF-HOME4/aia21.Aliases
PWF-SHARED	- 
fstype 
= 
ncp 
,ipserver 
= 
172.16.3.8 
,owner 
= 
aia21 
,uid 
= 
aia21 
,gid 
= 
aia21 
,hard 
,nodev 
,nosuid 
,nfsextras 
,strong,codepage=cp850,iocharset=utf8,filemode=640,dirmode=750,a=/ 
authcon/aia21	:PWF-SHARED/aia21.Aliases

On logout I then do this:

# Kill user's processes politely at first, then less so.
su -c "kill -TERM -1" $user
sleep 1
su -c "kill -KILL -1" $user
sleep 1
# For logging purposes
ps -U $user

# Tidy up the user's (auto)mounts.
(
         umount /home/$user
         umount /authcon/$user
         /sbin/killproc -USR1 /usr/sbin/automount
) >/dev/null 2>&1

And yes, this does expire everybody's non-busy mounts but that is not  
actually a bad thing.  They will get automounted on next use so there  
is no problem.  And it has the nice side effect that it will cause  
broken mounts to expire sooner.  (We have a recurring problem where  
people leaving themselves logged in for ages end up with broken  
connections and NCPfs does not support reconnects so this might  
actually help us.)

>>> Warning: before deploying a submount-based scheme in production,
>>> watch for
>>> a resolution of the internal locking (mutex) issue that's so
>>> bedeviling me.
>>
>> Could you elaborate on this?  What issue are you seeing?  Thanks a  
>> lot
>> in advance!
>
> Jim sees autofs hang.
> There is synchronization problem between submounts that are shutting
> down at the same time a mount request comes in. Of course we only  
> see it
> on busy systems.


I assume someone has attached to such a hung machine with gdb and  
gotten the two (or more) stack traces involved in the dead lock to  
find where in the code this happens?  If not, that begs the question  
as to why not?  (-:  Also if the problem really is with a mutex then  
the kernel's own mutex debugging code should pick up on the dead lock  
and produce stack traces, too.  Jim, have you tried running a kernel  
with lock debugging features turned on?

If there is a reproducible test case, perhaps a script that can be run  
to trigger it, I could look into it if no-one else has done it  
already...

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/

Re: Horrible denial of service bug in autmount 5

[Jim Carter]

On Tue, 24 Jun 2008, Anton Altaparmakov wrote:

> Ok.  I have for testing purposes done the changes you suggested and am using a
> file map successfully so we probably will not want/need to use a program map
> given the overhead it would impose.  Much easier to regenerate the file map
> for each user when they log in.  We now have:
> 
> /etc/auto.master:
> /.servers	file:/etc/auto.user
> 
> /etc/auto.user:
> * -fstype=autofs file:/var/run/pwfautomount/auto.&
> 
> And /var/run/pwfautomount/auto.<username> is generated when <username> user
> logs in and for my user aia21 contains exactly the same content as before:

This looks like a very sane way to handle your situation.  

> # Tidy up the user's (auto)mounts.
> (
>        umount /home/$user
>        umount /authcon/$user
>        /sbin/killproc -USR1 /usr/sbin/automount
> ) >/dev/null 2>&1
> 
> And yes, this does expire everybody's non-busy mounts but that is not actually
> a bad thing.  They will get automounted on next use so there is no problem.
> And it has the nice side effect that it will cause broken mounts to expire
> sooner.  (We have a recurring problem where people leaving themselves logged
> in for ages end up with broken connections and NCPfs does not support
> reconnects so this might actually help us.)

So it's a similar deal as CIFS: you need, and don't have, the user's 
plaintext password if the automounter ever expires the mount and then tries 
to reconnect when it's referred to again (in the same session).  Bummer.  
At least with CIFS there's the possibility of getting a Kerberos ticket 
valid on Windows at login time, and making it available via rpc.gssd or 
something like that.  I want to experiment with that but haven't 
been able to make time for it yet -- some of our users would really get 
good use out of such a feature.

It's much better if the filesystem is not mounted when the server or client 
crashes (or some idiot unplugs it).  Automounter expiration is good, if 
feasible for the filesystem type.  It also disposes of resources promptly 
without extra scripting by the sysop that could get broken.

> I assume someone has attached to such a hung machine with gdb and gotten the
> two (or more) stack traces involved in the dead lock to find where in the code
> this happens?  

More like 130 threads to be reported, in my case :-( Yes, I have this all 
automated.  You can take a look at some of the tracebacks in the mailing 
list archive (oink).  The rate of hanging seems to be proportional to the 
square of the rate of mount-expire cycles, being caused by a race condition 
getting a mutex, so it was noticed first on our webservers, which serve 
automounted UserDirs.  Ian Kent and I are making good progress stomping 
this bug (he fixes, I break it again :-).

James F. Carter          Voice 310 825 2897    FAX 310 206 6673
UCLA-Mathnet;  6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA 90095-1555
Email: jimc@math.ucla.edu  http://www.math.ucla.edu/~jimc (q.v. for PGP key)

Re: Horrible denial of service bug in autmount 5

[Jim Carter]

On Tue, 24 Jun 2008, Anton Altaparmakov wrote:

> >This problem is interesting: how would you do this while using the
> >multi-thread capability of autofs version 5?  Here's how far I got.  First,
> >we're not going to mount anything at login time; we're going to mount the
> >homedir the first time it's referenced, which on my system would be
> >slightly before logging in is finished (to check the ssh key in the
> >homedir).
> 
> Note this is not possible for us.  You cannot use SSH-keys when the home
> directory is on a Netware server because you need the user's password to be
> able to mount the home directory, there is no way to perform the mount before
> asking the user for their password.  

Oh, I didn't mean that you should use ssh authentication (in the sense that 
if the given password decrypts the ssh private key, that's sufficient to 
authenticate the user), I meant that we always load the keys into an agent 
process if existing, which requires automounting the homedir partway 
through PAM processing before the user is fully logged in.

Are you saying that publickey authentication in ssh is desired by users but 
is not feasible, because the plaintext password is required to mount the 
homedir to read the public key?  I guess that's right, when the homedir is 
on NCP or CIFS.  

> >Warning: before deploying a submount-based scheme in production, watch for
> >a resolution of the internal locking (mutex) issue that's so bedeviling me.
> 
> Could you elaborate on this?  What issue are you seeing?  Thanks a lot in
> advance!

Ian Kent described it pretty well in his reply.  The thread that's trying 
to mount a directory just hangs, and any further reference to that dir 
piles up waiting forever for the mount to finish.  At least the client 
process can be killed.

James F. Carter          Voice 310 825 2897    FAX 310 206 6673
UCLA-Mathnet;  6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA 90095-1555
Email: jimc@math.ucla.edu  http://www.math.ucla.edu/~jimc (q.v. for PGP key)

Re: Horrible denial of service bug in autmount 5

[Anton Altaparmakov]

Hi,

On 24 Jun 2008, at 19:55, Jim Carter wrote:
> On Tue, 24 Jun 2008, Anton Altaparmakov wrote:
>>> This problem is interesting: how would you do this while using the
>>> multi-thread capability of autofs version 5?  Here's how far I  
>>> got.  First,
>>> we're not going to mount anything at login time; we're going to  
>>> mount the
>>> homedir the first time it's referenced, which on my system would be
>>> slightly before logging in is finished (to check the ssh key in the
>>> homedir).
>>
>> Note this is not possible for us.  You cannot use SSH-keys when the  
>> home
>> directory is on a Netware server because you need the user's  
>> password to be
>> able to mount the home directory, there is no way to perform the  
>> mount before
>> asking the user for their password.
>
> Oh, I didn't mean that you should use ssh authentication (in the  
> sense that
> if the given password decrypts the ssh private key, that's  
> sufficient to
> authenticate the user), I meant that we always load the keys into an  
> agent
> process if existing, which requires automounting the homedir partway
> through PAM processing before the user is fully logged in.
>
> Are you saying that publickey authentication in ssh is desired by  
> users but
> is not feasible, because the plaintext password is required to mount  
> the
> homedir to read the public key?  I guess that's right, when the  
> homedir is
> on NCP or CIFS.

Yes, that is correct.  Users want it and we can't give it to them as  
we cannot read their ~/.ssh directory contents until we have asked for  
their plaintext password and used it to authenticate to the netware  
server.

Best regards,

	Anton

>>> Warning: before deploying a submount-based scheme in production,  
>>> watch for
>>> a resolution of the internal locking (mutex) issue that's so  
>>> bedeviling me.
>>
>> Could you elaborate on this?  What issue are you seeing?  Thanks a  
>> lot in
>> advance!
>
> Ian Kent described it pretty well in his reply.  The thread that's  
> trying
> to mount a directory just hangs, and any further reference to that dir
> piles up waiting forever for the mount to finish.  At least the client
> process can be killed.

-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/

Re: Horrible denial of service bug in autmount 5

[Ian Kent]

On Mon, 2008-06-23 at 03:26 +0100, J.P. King wrote:
> > Is there some reason you can't update the master map and send a HUP
> > signal to the daemon?
> 
> As I said previously, because this is quite hard.  The maps need to be 
> changed by the login process.  We currently do this in the PAM 
> configuration.  Given that we can have multiple log in at the same time 
> this means we need to do a lot of careful locking.

Yep.

> 
> Ideally we would do something like include all the files from a 
> configuration directory and have a HUP get it to re-read all these 
> configuration snippets.  This would be a relatively involved change 
> however.

The daemon reads the master map to find out what to needs to change, so
you would still need to update the master map.

I understand that largish changes aren't good and there's the
possibility of clients running different versions of autofs.

> 
> Please do remember where we came in though - we have our needs, which is 
> our problem.  You have a bug/DOS which we identified whilst looking for a 
> reason that something had broken between OpenSUSE 10.2 and OpenSUSE 10.3.
> Clearly we are going to hope that when you fix your bug we can backport 
> the solution and solve our problem.  If we can't we can't, and we will 
> work around whatever issues we have left.

You keep on about this, I've already said I'll fix it, let it go!
I'll add an option to override the behavior and change the check in line
with Peter's suggestion. But, being able to say that running multiple
instances is supported is a completely different issue which will
require more effort.

Ian

Re: Horrible denial of service bug in autmount 5

[J.P. King]

>> our problem.  You have a bug/DOS which we identified whilst looking for a
>> reason that something had broken between OpenSUSE 10.2 and OpenSUSE 10.3.
>> Clearly we are going to hope that when you fix your bug we can backport
>> the solution and solve our problem.  If we can't we can't, and we will
>> work around whatever issues we have left.
>
> You keep on about this, I've already said I'll fix it, let it go!

I'm trying to emphasise that we aren't demanding that you make changes to 
meet our needs!  I don't want anyone reading this thread to think that 
we're throwing our toys out of the pram demanding that changes be made to 
suit our needs.

> Ian

Julian
--
Julian King
Computer Officer, University of Cambridge, Unix Support

Re: Horrible denial of service bug in autmount 5

[Ian Kent]

On Mon, 2008-06-23 at 03:55 +0100, J.P. King wrote:
> >> our problem.  You have a bug/DOS which we identified whilst looking for a
> >> reason that something had broken between OpenSUSE 10.2 and OpenSUSE 10.3.
> >> Clearly we are going to hope that when you fix your bug we can backport
> >> the solution and solve our problem.  If we can't we can't, and we will
> >> work around whatever issues we have left.
> >
> > You keep on about this, I've already said I'll fix it, let it go!
> 
> I'm trying to emphasise that we aren't demanding that you make changes to 
> meet our needs!  I don't want anyone reading this thread to think that 
> we're throwing our toys out of the pram demanding that changes be made to 
> suit our needs.

I see, btw, what is the OpenSuSE rpm revision you are working with?
I see an rpm of autofs-5.0.2-30.2.i586.rpm and I'm assuming the
autofs.rpm is the source corresponding to that or am I missing something
about the OpenSuSE packaging.

Ian

Re: Horrible denial of service bug in autmount 5

[Anton Altaparmakov]

Good morning,

On 23 Jun 2008, at 04:10, Ian Kent wrote:
> I see, btw, what is the OpenSuSE rpm revision you are working with?
> I see an rpm of autofs-5.0.2-30.2.i586.rpm and I'm assuming the
> autofs.rpm is the source corresponding to that or am I missing  
> something
> about the OpenSuSE packaging.


The one we are using is indeed autofs-5.0.2-30.2.i586.rpm and the  
corresponding source rpm is autofs-5.0.2-30.2.src.rpm and it can be  
obtained for example here:

	http://www.mirrorservice.org/sites/ftp.opensuse.org/pub/opensuse/update/10.3/rpm/src/autofs-5.0.2-30.2.src.rpm

Regarding the need to run a single instance: you do realise that init  
scripts do the exclusion already?  So there should not be any reason  
to repeat the exclusion in the daemon itself...  At least on SuSE  
systems this is done by executing daemons using "/sbin/startproc"  
rather than running the daemons themselves.  So you are basically  
repeating everything the init script is already doing in the automount  
daemon which IMHO is rather pointless given I assume all Linux  
flavours use an analogous init script to launch the automount daemon.

I have included below the startproc man page if you want to have a  
look what it does...

And for people like us who are not using an init script you can safely  
assume we know what we are doing and that we are invoking the  
automount daemon multiple times because we want to rather than because  
we are stupid and need to be protected from ourselves.  (-:

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/


<quote "man startproc">
STARTPROC(8)                 The SuSE boot concept                 
STARTPROC(8)



NAME
        Startproc - Start processes identified by path name
        Start_daemon - Start processes identified by path name

SYNOPSIS
        startproc [-f] [-L] [[-n ]+/-<prio>] [-s] [-t sec] [-u user] [- 
g group]
        [-v] [-e] [-l log_file |-q|-d] [-p pid_file] [-i ignore_file]  
[-c root]
        /path/to/executable [arguments for executable]

        start_daemon  [-f]  [-n  +/-<prio>]  /path/to/executable  
[arguments for
        executable]

DESCRIPTION
        startproc and the LSB variant start_daemon check for all   
processes  of
        the  specified executable and starts it if no processes are  
found. Note
        that startproc is designed to start a daemon but not a kernel  
thread or
        a program which enables a kernel thread.

        startproc  does  not  use  the pid to search for a process but  
the full
        path of the corresponding program which is used to  identify   
the  exe‐
        cutable  (see proc(5)).  Only if the inode number (/proc/<pid>/ 
exe) and
        the full name are unavailable  (/proc/<pid>/cmdline)  or  if   
the  exe‐
        cutable  changes  its  zeroth  argument,  startproc  uses the  
base name
        (/proc/<pid>/stat) to identify the running program.

        Extended functionality is provided by the -p option (former   
option  -f
        changed  due  to  the LSB specification).  If this option is  
specified,
        startproc tries to check against the pid read from this file  
instead of
        the  default (/var/run/<basename>.pid).  The pid read from  
this file is
        compared against the pids of possible running processes  that   
use  the
        specified executable. In order to avoid confusion with stale  
pid files,
        a not up-to-date pid will be ignored.

        For the possibility of having two different sessions of one  
binary pro‐
        gram,  the option -i ignore_file allows to specify a pid file  
which pid
        number is used to ignore all processes of  corresponding   
process  ses‐
        sion.

        The option -v makes startproc print out verbose diagnostics.

REQUIRED
        /path/to/executable
               Specifies the executable by its full path name. This  
argument is
               always required. Everything that follows this path is  
considered
               options  for  the executable to be launched. Further  
information
               can be obtained from the respective manpage(s).


OPTIONS
        [-n ]+/-<prio>
               Set the nice level for the process. If used, this  
option  should
               always  be the first in the command line.  The nice  
level <prio>
               may be specified in the range between -20 and +20.   
Only root is
               allowed to set negative nice values.

        -e     Bequeath  only a minimal set of environment variables  
to the new
               process: HOME, PATH, SHELL, RUNLEVEL, and PREVLEVEL.

        -p pid_file
               (Former option -f changed due to the LSB  
specification.)  Use an
               alternate  pid  file  instead  of  the  default (/var/ 
run/<base‐
               name>.pid).  The pid  read  from  this  file  is   
being  matched
               against  the  pid  of  running processes that have an  
executable
               with specified path.  of the program.  In order to  
avoid  confu‐
               sion with stale pid files, a not up-to-date pid will be  
ignored.

        -i ignore_file
               The pid found in this file is used as session  id  of   
the  same
               binary program which should be ignored by startproc.

        -f     This option is required by the Linux Standard Base  
Specification
               (LSB).  With this option the start of a process is  
forced.

        -g group
               Sets the group ID of the process to gid.

        -l log_file
               Redirect the process standard output and standard  
error  to  the
               file log_file.

        -L     This  option  causes  symlinks to be followed, as the  
like-named
               option in ls(1).  BR Note : for the file name the  
original  name
               of the program is used instead of the name of the  
symbolic link.

        -c root
               Change root directory to root.  Services which have  
been started
               with  this  option  can only be checked by checkproc(8)  
and sig‐
               naled by killproc(8) if checkproc(8) and killproc(8)  
are  called
               with the same option argument for the option -c.

        -q     Equals to -l /dev/null (supresses output).

        -d     Let  startproc  expect that the started service will do  
a dialog
               by prompting for, e.g. a  passphrase.   This  option   
implies  a
               timeout of 15 seconds (-t 15).

        -s     Starts  the  process in a new session. The new task is  
a process
               group leader and has no controlling tty.

        -t sec The number of seconds to wait after the successful   
start  of  a
               service.   This  options  accepts the number of seconds  
to wait.
               You can specify some units after a given number: s for   
seconds,
               m for minutes, and h for hours to wait.

        -u user
               Sets the user ID of the process to user.

        -v     Verbose output.

EXAMPLE
        startproc /usr/sbin/sendmail

               starts  /usr/sbin/sendmail if no sendmail process is  
found. If a
               pid file sendmail.pid exists in /var/run/, then the pid  
found in
               this file is used to search the process table for a  
process with
               an   executable   that   matches   the    specified     
pathname,
               /usr/sbin/sendmail.    If   no   matching   process    
is  found,
               /usr/sbin/sendmail is launched.

        startproc -p /var/myrun/lpd.pid /usr/sbin/lpd

               starts /usr/sbin/lpd if there is no process with the   
pid  found
               in /var/myrun/lpd.pid and no process in the actual  
process table
               exists that uses the specified binary.

EXIT CODES
        The exit codes have the following LSB conform conditions:


               0    Success

               1    Generic or unspecified error

               2    Invalid or excess argument(s)

               4    Insufficient privilege(s)

               5    Program is not installed

               7    Program is not running

        In some error cases, diagnostic output is sent to standard   
error,  or,
        if standard error is not available, syslogd(8) is being used.

NOTE
        startproc  is  a replacement for the Bourne shell function  
daemon found
        in  the  widely  used  SysVinit  package  of  Miquel  van   
Smoorenburg,
        <miquels@cistron.nl>.  startproc is not useful to start kernel  
threads.
        This should be done by service utilities designed for  the   
purpose  to
        accomplish this task.

BUGS
        Identifying  a process based on the executable file and the  
correspond‐
        ing inode number only works if the process stays  alive   
during  start‐
        proc's  execution.  Processes  rewriting their zeroth argument  
or shell
        scripts (the inode number of the shell executable file is not  
identical
        to that of the script file) may not be identified by a  
filename path.

        Startproc does not start a process if there already exists one  
being in
        the zombie state.  Zombies are processes which arn't alive   
but  listed
        in  the process table to have the exit status ready for the  
correspond‐
        ing parent processes.  Therefore the parent processes should   
be  check
        out.

FILES
        /proc/ path to the proc file system (see proc(5)).

        /etc/init.d/
               path  to the SuSE boot concept script base directory as  
required
               by  the  Linux  Standard    Base    Specification    
(LSB)   (see
               init.d(7)).

SEE ALSO
        checkproc(8),  killproc(8),  insserv(8),  init.d(7), kill(1),  
skill(1),
        killall(8), killall5(8), signal(7), proc(5).

COPYRIGHT
        1994-2000 Werner Fink, 1996-2000 SuSE GmbH Nuernberg, Germany.

AUTHOR
        Werner Fink <werner@suse.de>



3rd Berkeley Distribution        Nov 10, 2000                      
STARTPROC(8)
</quote>

_______________________________________________
autofs mailing list
autofs@linux.kernel.org
http://linux.kernel.org/mailman/listinfo/autofs

Re: Horrible denial of service bug in autmount 5

[Ian Kent]

On Mon, 2008-06-23 at 08:15 +0100, Anton Altaparmakov wrote:
> Good morning,
> 
> On 23 Jun 2008, at 04:10, Ian Kent wrote:
> > I see, btw, what is the OpenSuSE rpm revision you are working with?
> > I see an rpm of autofs-5.0.2-30.2.i586.rpm and I'm assuming the
> > autofs.rpm is the source corresponding to that or am I missing  
> > something
> > about the OpenSuSE packaging.
> 
> 
> The one we are using is indeed autofs-5.0.2-30.2.i586.rpm and the  
> corresponding source rpm is autofs-5.0.2-30.2.src.rpm and it can be  
> obtained for example here:
> 
> 	http://www.mirrorservice.org/sites/ftp.opensuse.org/pub/opensuse/update/10.3/rpm/src/autofs-5.0.2-30.2.src.rpm

Got it.
Will check patches against it and make any changes needed.

> 
> Regarding the need to run a single instance: you do realise that init  
> scripts do the exclusion already?  So there should not be any reason  
> to repeat the exclusion in the daemon itself...  At least on SuSE  
> systems this is done by executing daemons using "/sbin/startproc"  
> rather than running the daemons themselves.  So you are basically  
> repeating everything the init script is already doing in the automount  
> daemon which IMHO is rather pointless given I assume all Linux  
> flavours use an analogous init script to launch the automount daemon.

Yeah, not all distributions use a function like this.

The idea here is that, if someone runs the program directly from the
command line, I want the same behavior except for possibly command line
arguments present in the configuration. Such as if someone runs the
program in the foreground maybe with debug enabled. A check to see if it
is already running is just a small part of that.

Ian

Re: Horrible denial of service bug in autmount 5

[Jim Carter]

On Mon, 23 Jun 2008, Ian Kent wrote:

> Is there some reason you can't update the master map and send a HUP
> signal to the daemon?

Actually we have a (simpler) setup: the master map refers to a NIS map for 
home directories (would be LDAP on a non-Solaris-derived installation).  
This map is rebuilt whenever a user is added, removed or relocated, and 
automount sees the change for $USER at the next attempt to mount 
/home/$USER.

When users have special file storage they refer to /net/$HOST/$filesys
(example: /net/julia/h1) and whatever path components are below it.  If 
it's not acceptable to hard-code the host (and with 30,000 users where 
potentially all of them need to do this, likely hardcoded hostnames and 
partitions are not cool), we would create another NIS or LDAP map analogous 
to auto.home, and update and republish it whenever a user's special storage 
was altered.

James F. Carter          Voice 310 825 2897    FAX 310 206 6673
UCLA-Mathnet;  6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA 90095-1555
Email: jimc@math.ucla.edu  http://www.math.ucla.edu/~jimc (q.v. for PGP key)