[PATCH 0/3] Various gssd fixes including machine-credential issue.

[PATCH 0/3] Various gssd fixes including machine-credential issue.

[Neil Brown]

As you probably know, since 3.7 (I think) Linux NFS has explicitly
asked for machine credentials for certain requests rather than asking
for root credentials as is previously did.
This causes a regression for people who don't have any machine
credentials configured and use "gssd -n".

I gather this was discussed on the mailing list earlier this year but
not resolved.

I would like to re-awaken the issue and offer a resolution (which has
been tested and found effective by a customer).

Hence these three patches.  The first two are minor issues that I
stumbled over while trying to understand the problem and are not
critical but probably should be fixed.

The third addresses the above mentioned issue.  It introduces a
variable "machine_uses_root_credentials" which is similar to the
current "root_uses_machine_credentials".  It also adds a "-N" flag to
set this variable.

I'm not certain what the defaults should be.  For backward
compatibility it would be best if '-n' set the this new variable as
well as clearing the old one, but then I'm not sure what exactly -N
should do.

Comments welcome.

Thanks,
NeilBrown



---

Neil Brown (3):
      krb5_utils: remove redundant array size.
      krb5_util: don't give up on machine credential if hostname not available.
      gssd: add -N option to use root credentials as machine credentials.


 utils/gssd/gssd.c      |    9 ++++++---
 utils/gssd/gssd.h      |    1 +
 utils/gssd/gssd.man    |   13 ++++++++++++-
 utils/gssd/gssd_proc.c |   12 +++++++-----
 utils/gssd/krb5_util.c |   10 +++++++---
 5 files changed, 33 insertions(+), 12 deletions(-)

-- 
Signature

[PATCH 1/3] krb5_utils: remove redundant array size.

[Neil Brown]

When initialising an array there is no need to specify the size as the
size is taken from the initialiser.  Having the size there means that
any change to the initialiser needs to change the size to and so is
error-prone.

So just remove the size.

Signed-off-by: NeilBrown <neilb@suse.de>
---
 utils/gssd/krb5_util.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/utils/gssd/krb5_util.c b/utils/gssd/krb5_util.c
index 6275dd8..9ef80f0 100644
--- a/utils/gssd/krb5_util.c
+++ b/utils/gssd/krb5_util.c
@@ -1236,7 +1236,7 @@ gssd_refresh_krb5_machine_credential(char *hostname,
 	krb5_keytab kt = NULL;;
 	int retval = 0;
 	char *k5err = NULL;
-	const char *svcnames[5] = { "$", "root", "nfs", "host", NULL };
+	const char *svcnames[] = { "$", "root", "nfs", "host", NULL };
 
 	/*
 	 * If a specific service name was specified, use it.

Re: [PATCH 1/3] krb5_utils: remove redundant array size.

[Steve Dickson]

On 02/06/13 21:00, Neil Brown wrote:
> When initialising an array there is no need to specify the size as the
> size is taken from the initialiser.  Having the size there means that
> any change to the initialiser needs to change the size to and so is
> error-prone.
> 
> So just remove the size.
Committed....

steved.
> 
> Signed-off-by: NeilBrown <neilb@suse.de>
> ---
>  utils/gssd/krb5_util.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/utils/gssd/krb5_util.c b/utils/gssd/krb5_util.c
> index 6275dd8..9ef80f0 100644
> --- a/utils/gssd/krb5_util.c
> +++ b/utils/gssd/krb5_util.c
> @@ -1236,7 +1236,7 @@ gssd_refresh_krb5_machine_credential(char *hostname,
>  	krb5_keytab kt = NULL;;
>  	int retval = 0;
>  	char *k5err = NULL;
> -	const char *svcnames[5] = { "$", "root", "nfs", "host", NULL };
> +	const char *svcnames[] = { "$", "root", "nfs", "host", NULL };
>  
>  	/*
>  	 * If a specific service name was specified, use it.
> 
> 

[PATCH 3/3] gssd: add -N option to use root credentials as machine credentials.

[Neil Brown]

Since linux-3.7, the kernel asks explicitly for machine credentials
rather than root credentials to authenticate state management requests.

This causes a regression for people who do not have machine
credentials configured and were using "gssd -n" to instruct gssd to
disable the default mapping of using machine credentials to authorise
accesses by 'root'.

This patch adds '-N' flag which instruct gssd explicitly to use 'root'
credentials whenever 'machine' credentials are requested.  Thus
  gssd -n -N
provides the same service that
  gssd -n
used to.

In summary:

Credentials used for different request types and different gssd flags:

   Request type: |     "gssd"      "gssd -n"   "gssd -N"    "gssd -nN"
                 |
     machine     |     machine      machine     root         root
                 |
     root        |     machine      root        machine      root

Signed-off-by: NeilBrown <neilb@suse.de>
---
 utils/gssd/gssd.c      |    9 ++++++---
 utils/gssd/gssd.h      |    1 +
 utils/gssd/gssd.man    |   13 ++++++++++++-
 utils/gssd/gssd_proc.c |   12 +++++++-----
 4 files changed, 26 insertions(+), 9 deletions(-)

diff --git a/utils/gssd/gssd.c b/utils/gssd/gssd.c
index 8ee478b..7a405b6 100644
--- a/utils/gssd/gssd.c
+++ b/utils/gssd/gssd.c
@@ -61,6 +61,7 @@ char ccachedir[PATH_MAX] = GSSD_DEFAULT_CRED_DIR ":" GSSD_USER_CRED_DIR;
 char *ccachesearch[GSSD_MAX_CCACHE_SEARCH + 1];
 int  use_memcache = 0;
 int  root_uses_machine_creds = 1;
+int  machine_uses_root_creds = 0;
 unsigned int  context_timeout = 0;
 char *preferred_realm = NULL;
 
@@ -68,8 +69,7 @@ void
 sig_die(int signal)
 {
 	/* destroy krb5 machine creds */
-	if (root_uses_machine_creds)
-		gssd_destroy_krb5_machine_creds();
+	gssd_destroy_krb5_machine_creds();
 	printerr(1, "exiting on signal %d\n", signal);
 	exit(0);
 }
@@ -102,7 +102,7 @@ main(int argc, char *argv[])
 	char *progname;
 
 	memset(ccachesearch, 0, sizeof(ccachesearch));
-	while ((opt = getopt(argc, argv, "DfvrlmnMp:k:d:t:R:")) != -1) {
+	while ((opt = getopt(argc, argv, "DfvrlmNnMp:k:d:t:R:")) != -1) {
 		switch (opt) {
 			case 'f':
 				fg = 1;
@@ -116,6 +116,9 @@ main(int argc, char *argv[])
 			case 'n':
 				root_uses_machine_creds = 0;
 				break;
+			case 'N':
+				machine_uses_root_creds = 1;
+				break;
 			case 'v':
 				verbosity++;
 				break;
diff --git a/utils/gssd/gssd.h b/utils/gssd/gssd.h
index 86472a1..5057440 100644
--- a/utils/gssd/gssd.h
+++ b/utils/gssd/gssd.h
@@ -65,6 +65,7 @@ extern char			keytabfile[PATH_MAX];
 extern char			*ccachesearch[];
 extern int			use_memcache;
 extern int			root_uses_machine_creds;
+extern int			machine_uses_root_creds;
 extern unsigned int 		context_timeout;
 extern char			*preferred_realm;
 
diff --git a/utils/gssd/gssd.man b/utils/gssd/gssd.man
index ac13fd4..0a06e8c 100644
--- a/utils/gssd/gssd.man
+++ b/utils/gssd/gssd.man
@@ -8,7 +8,7 @@
 rpc.gssd \- RPCSEC_GSS daemon
 .SH SYNOPSIS
 .B rpc.gssd
-.RB [ \-DfMnlvr ]
+.RB [ \-DfMnNlvr ]
 .RB [ \-k
 .IR keytab ]
 .RB [ \-p
@@ -227,6 +227,17 @@ in the foreground and sends output to stderr (as opposed to syslogd)
 When specified, UID 0 is forced to obtain user credentials
 which are used instead of the local system's machine credentials.
 .TP
+.B -N
+With NFSv4, some requests to the server need to authenticated
+as coming from "the machine" rather than from any particular user.
+These requests will normally be authenticated using the "machine
+credentials" even if
+.B -n
+is set.  Adding
+.B -N
+causes these requests to use the credentials of UID 0 in place of the
+machine credentials.
+.TP
 .BI "-k " keytab
 Tells
 .B rpc.gssd
diff --git a/utils/gssd/gssd_proc.c b/utils/gssd/gssd_proc.c
index b7e2bbb..f9d6f51 100644
--- a/utils/gssd/gssd_proc.c
+++ b/utils/gssd/gssd_proc.c
@@ -988,7 +988,8 @@ process_krb5_upcall(struct clnt_info *clp, uid_t uid, int fd, char *tgtname,
 	/*
 	 * If "service" is specified, then the kernel is indicating that
 	 * we must use machine credentials for this request.  (Regardless
-	 * of the uid value or the setting of root_uses_machine_creds.)
+	 * of the uid value or the setting of root_uses_machine_creds,
+	 * though setting machine_uses_root_creds can override this)
 	 * If the service value is "*", then any service name can be used.
 	 * Otherwise, it specifies the service name that should be used.
 	 * (For now, the values of service will only be "*" or "nfs".)
@@ -1008,8 +1009,9 @@ process_krb5_upcall(struct clnt_info *clp, uid_t uid, int fd, char *tgtname,
 	 */
 	printerr(2, "%s: service is '%s'\n", __func__,
 		 service ? service : "<null>");
-	if (uid != 0 || (uid == 0 && root_uses_machine_creds == 0 &&
-				service == NULL)) {
+	if (uid != 0 ||
+	    (!root_uses_machine_creds && !service) ||
+	    ( machine_uses_root_creds &&  service)) {
 		/* Tell krb5 gss which credentials cache to use */
 		/* Try first to acquire credentials directly via GSSAPI */
 		err = gssd_acquire_user_cred(uid, &gss_cred);
@@ -1028,8 +1030,8 @@ process_krb5_upcall(struct clnt_info *clp, uid_t uid, int fd, char *tgtname,
 		}
 	}
 	if (create_resp != 0) {
-		if (uid == 0 && (root_uses_machine_creds == 1 ||
-				service != NULL)) {
+		if ((uid == 0 && root_uses_machine_creds) ||
+		    (service != NULL && !machine_uses_root_creds)) {
 			int nocache = 0;
 			int success = 0;
 			do {

Re: [PATCH 3/3] gssd: add -N option to use root credentials as machine credentials.

[Steve Dickson]

Neil,

On 02/06/13 21:00, Neil Brown wrote:
> Since linux-3.7, the kernel asks explicitly for machine credentials
> rather than root credentials to authenticate state management requests.
> 
> This causes a regression for people who do not have machine
> credentials configured and were using "gssd -n" to instruct gssd to
> disable the default mapping of using machine credentials to authorise
> accesses by 'root'.
> 
> This patch adds '-N' flag which instruct gssd explicitly to use 'root'
> credentials whenever 'machine' credentials are requested.  Thus
>   gssd -n -N
> provides the same service that
>   gssd -n
> used to.
> 
> In summary:
> 
> Credentials used for different request types and different gssd flags:
> 
>    Request type: |     "gssd"      "gssd -n"   "gssd -N"    "gssd -nN"
>                  |
>      machine     |     machine      machine     root         root
>                  |
>      root        |     machine      root        machine      root
> 
> Signed-off-by: NeilBrown <neilb@suse.de>
So is this no longer needed do the kernel change you and Chuck came up with?

steved.

Re: [PATCH 3/3] gssd: add -N option to use root credentials as machine credentials.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1359 bytes --]

On Mon, 01 Jul 2013 12:23:22 -0400 Steve Dickson <SteveD@redhat.com> wrote:

> Neil,
> 
> On 02/06/13 21:00, Neil Brown wrote:
> > Since linux-3.7, the kernel asks explicitly for machine credentials
> > rather than root credentials to authenticate state management requests.
> > 
> > This causes a regression for people who do not have machine
> > credentials configured and were using "gssd -n" to instruct gssd to
> > disable the default mapping of using machine credentials to authorise
> > accesses by 'root'.
> > 
> > This patch adds '-N' flag which instruct gssd explicitly to use 'root'
> > credentials whenever 'machine' credentials are requested.  Thus
> >   gssd -n -N
> > provides the same service that
> >   gssd -n
> > used to.
> > 
> > In summary:
> > 
> > Credentials used for different request types and different gssd flags:
> > 
> >    Request type: |     "gssd"      "gssd -n"   "gssd -N"    "gssd -nN"
> >                  |
> >      machine     |     machine      machine     root         root
> >                  |
> >      root        |     machine      root        machine      root
> > 
> > Signed-off-by: NeilBrown <neilb@suse.de>
> So is this no longer needed do the kernel change you and Chuck came up with?
> 
> steved.

That is correct.  This patch is no longer needed.

Thanks,
NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

[PATCH 2/3] krb5_util: don't give up on machine credential if hostname not available.

[Neil Brown]

krb5_util tries various different credential names in order to find
the machine credential, not all of them use the full host name of the
current host.

So if getting the full host name fails, don't give up completely,
still try the other options.

Signed-off-by: NeilBrown <neilb@suse.de>
---
 utils/gssd/krb5_util.c |    8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/utils/gssd/krb5_util.c b/utils/gssd/krb5_util.c
index 9ef80f0..5e84481 100644
--- a/utils/gssd/krb5_util.c
+++ b/utils/gssd/krb5_util.c
@@ -825,8 +825,10 @@ find_keytab_entry(krb5_context context, krb5_keytab kt, const char *tgtname,
 	myhostad[i+1] = 0;
 
 	retval = get_full_hostname(myhostname, myhostname, sizeof(myhostname));
-	if (retval)
-		goto out;
+	if (retval) {
+		/* Don't use myhostname */
+		myhostname[0] = 0;
+	}
 
 	code = krb5_get_default_realm(context, &default_realm);
 	if (code) {
@@ -883,6 +885,8 @@ find_keytab_entry(krb5_context context, krb5_keytab kt, const char *tgtname,
 								myhostad,
 								NULL);
 			} else {
+				if (!myhostname[0])
+					continue;
 				snprintf(spn, sizeof(spn), "%s/%s@%s",
 					 svcnames[j], myhostname, realm);
 				code = krb5_build_principal_ext(context, &princ,

Re: [PATCH 2/3] krb5_util: don't give up on machine credential if hostname not available.

[Steve Dickson]

Sorry for getting into so late... I did an extraordinary amount
of travailing in June.... 

On 02/06/13 21:00, Neil Brown wrote:
> krb5_util tries various different credential names in order to find
> the machine credential, not all of them use the full host name of the
> current host.
> 
> So if getting the full host name fails, don't give up completely,
> still try the other options.
> 
> Signed-off-by: NeilBrown <neilb@suse.de>
> ---
>  utils/gssd/krb5_util.c |    8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/utils/gssd/krb5_util.c b/utils/gssd/krb5_util.c
> index 9ef80f0..5e84481 100644
> --- a/utils/gssd/krb5_util.c
> +++ b/utils/gssd/krb5_util.c
> @@ -825,8 +825,10 @@ find_keytab_entry(krb5_context context, krb5_keytab kt, const char *tgtname,
>  	myhostad[i+1] = 0;
>  
>  	retval = get_full_hostname(myhostname, myhostname, sizeof(myhostname));
> -	if (retval)
> -		goto out;
> +	if (retval) {
> +		/* Don't use myhostname */
> +		myhostname[0] = 0;
> +	}
>  
>  	code = krb5_get_default_realm(context, &default_realm);
>  	if (code) {
> @@ -883,6 +885,8 @@ find_keytab_entry(krb5_context context, krb5_keytab kt, const char *tgtname,
>  								myhostad,
>  								NULL);
>  			} else {
> +				if (!myhostname[0])
> +					continue;
>  				snprintf(spn, sizeof(spn), "%s/%s@%s",
>  					 svcnames[j], myhostname, realm);
>  				code = krb5_build_principal_ext(context, &princ,
> 
> 
At the end of day... This patch allows the machine cred to be used when
there is no DNS or /etc/hosts is empty (aka getaddrinfo() fails via 
the get_full_hostname() call).

I'm thinking this is a good idea, but I'm a gnawing feeling this would
be open some type of security hole by using machine creds when they
should not be or they were not expected to be used...

Am I being too paranoid???

steved.

Re: [PATCH 2/3] krb5_util: don't give up on machine credential if hostname not available.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2529 bytes --]

On Mon, 01 Jul 2013 12:22:22 -0400 Steve Dickson <SteveD@redhat.com> wrote:

> Sorry for getting into so late... I did an extraordinary amount
> of travailing in June.... 
> 
> On 02/06/13 21:00, Neil Brown wrote:
> > krb5_util tries various different credential names in order to find
> > the machine credential, not all of them use the full host name of the
> > current host.
> > 
> > So if getting the full host name fails, don't give up completely,
> > still try the other options.
> > 
> > Signed-off-by: NeilBrown <neilb@suse.de>
> > ---
> >  utils/gssd/krb5_util.c |    8 ++++++--
> >  1 file changed, 6 insertions(+), 2 deletions(-)
> > 
> > diff --git a/utils/gssd/krb5_util.c b/utils/gssd/krb5_util.c
> > index 9ef80f0..5e84481 100644
> > --- a/utils/gssd/krb5_util.c
> > +++ b/utils/gssd/krb5_util.c
> > @@ -825,8 +825,10 @@ find_keytab_entry(krb5_context context, krb5_keytab kt, const char *tgtname,
> >  	myhostad[i+1] = 0;
> >  
> >  	retval = get_full_hostname(myhostname, myhostname, sizeof(myhostname));
> > -	if (retval)
> > -		goto out;
> > +	if (retval) {
> > +		/* Don't use myhostname */
> > +		myhostname[0] = 0;
> > +	}
> >  
> >  	code = krb5_get_default_realm(context, &default_realm);
> >  	if (code) {
> > @@ -883,6 +885,8 @@ find_keytab_entry(krb5_context context, krb5_keytab kt, const char *tgtname,
> >  								myhostad,
> >  								NULL);
> >  			} else {
> > +				if (!myhostname[0])
> > +					continue;
> >  				snprintf(spn, sizeof(spn), "%s/%s@%s",
> >  					 svcnames[j], myhostname, realm);
> >  				code = krb5_build_principal_ext(context, &princ,
> > 
> > 
> At the end of day... This patch allows the machine cred to be used when
> there is no DNS or /etc/hosts is empty (aka getaddrinfo() fails via 
> the get_full_hostname() call).
> 
> I'm thinking this is a good idea, but I'm a gnawing feeling this would
> be open some type of security hole by using machine creds when they
> should not be or they were not expected to be used...
> 
> Am I being too paranoid???

Probably, but it is a good default position nonetheless.

This patch will only allow a machine credential to be used in the absence of
an easily detected "full hostname" if a wild card machine credential is
available.  And if such is available, it seems wrong not to use it.

If wildcard machine credentials were no expected to be used, it we seem
strange to have them included in the keytab file.

So I cannot see any hole.

Thanks,
NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [PATCH 2/3] krb5_util: don't give up on machine credential if hostname not available.

[Steve Dickson]

On 01/07/13 17:56, NeilBrown wrote:
> On Mon, 01 Jul 2013 12:22:22 -0400 Steve Dickson <SteveD@redhat.com> wrote:
> 
>> Sorry for getting into so late... I did an extraordinary amount
>> of travailing in June.... 
>>
>> On 02/06/13 21:00, Neil Brown wrote:
>>> krb5_util tries various different credential names in order to find
>>> the machine credential, not all of them use the full host name of the
>>> current host.
>>>
>>> So if getting the full host name fails, don't give up completely,
>>> still try the other options.
>>>
>>> Signed-off-by: NeilBrown <neilb@suse.de>
>>> ---
>>>  utils/gssd/krb5_util.c |    8 ++++++--
>>>  1 file changed, 6 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/utils/gssd/krb5_util.c b/utils/gssd/krb5_util.c
>>> index 9ef80f0..5e84481 100644
>>> --- a/utils/gssd/krb5_util.c
>>> +++ b/utils/gssd/krb5_util.c
>>> @@ -825,8 +825,10 @@ find_keytab_entry(krb5_context context, krb5_keytab kt, const char *tgtname,
>>>  	myhostad[i+1] = 0;
>>>  
>>>  	retval = get_full_hostname(myhostname, myhostname, sizeof(myhostname));
>>> -	if (retval)
>>> -		goto out;
>>> +	if (retval) {
>>> +		/* Don't use myhostname */
>>> +		myhostname[0] = 0;
>>> +	}
>>>  
>>>  	code = krb5_get_default_realm(context, &default_realm);
>>>  	if (code) {
>>> @@ -883,6 +885,8 @@ find_keytab_entry(krb5_context context, krb5_keytab kt, const char *tgtname,
>>>  								myhostad,
>>>  								NULL);
>>>  			} else {
>>> +				if (!myhostname[0])
>>> +					continue;
>>>  				snprintf(spn, sizeof(spn), "%s/%s@%s",
>>>  					 svcnames[j], myhostname, realm);
>>>  				code = krb5_build_principal_ext(context, &princ,
>>>
>>>
>> At the end of day... This patch allows the machine cred to be used when
>> there is no DNS or /etc/hosts is empty (aka getaddrinfo() fails via 
>> the get_full_hostname() call).
>>
>> I'm thinking this is a good idea, but I'm a gnawing feeling this would
>> be open some type of security hole by using machine creds when they
>> should not be or they were not expected to be used...
>>
>> Am I being too paranoid???
> 
> Probably, but it is a good default position nonetheless.
> 
> This patch will only allow a machine credential to be used in the absence of
> an easily detected "full hostname" if a wild card machine credential is
> available.  And if such is available, it seems wrong not to use it.
> 
> If wildcard machine credentials were no expected to be used, it we seem
> strange to have them included in the keytab file.
> 
> So I cannot see any hole.
Fair enough... Thanks! 

steved.

Re: [PATCH 2/3] krb5_util: don't give up on machine credential if hostname not available.

[Steve Dickson]

On 02/06/13 21:00, Neil Brown wrote:
> krb5_util tries various different credential names in order to find
> the machine credential, not all of them use the full host name of the
> current host.
> 
> So if getting the full host name fails, don't give up completely,
> still try the other options.
> 
> Signed-off-by: NeilBrown <neilb@suse.de>
Committed! 

steved.

> ---
>  utils/gssd/krb5_util.c |    8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/utils/gssd/krb5_util.c b/utils/gssd/krb5_util.c
> index 9ef80f0..5e84481 100644
> --- a/utils/gssd/krb5_util.c
> +++ b/utils/gssd/krb5_util.c
> @@ -825,8 +825,10 @@ find_keytab_entry(krb5_context context, krb5_keytab kt, const char *tgtname,
>  	myhostad[i+1] = 0;
>  
>  	retval = get_full_hostname(myhostname, myhostname, sizeof(myhostname));
> -	if (retval)
> -		goto out;
> +	if (retval) {
> +		/* Don't use myhostname */
> +		myhostname[0] = 0;
> +	}
>  
>  	code = krb5_get_default_realm(context, &default_realm);
>  	if (code) {
> @@ -883,6 +885,8 @@ find_keytab_entry(krb5_context context, krb5_keytab kt, const char *tgtname,
>  								myhostad,
>  								NULL);
>  			} else {
> +				if (!myhostname[0])
> +					continue;
>  				snprintf(spn, sizeof(spn), "%s/%s@%s",
>  					 svcnames[j], myhostname, realm);
>  				code = krb5_build_principal_ext(context, &princ,
> 
> 

Re: [PATCH 0/3] Various gssd fixes including machine-credential issue.

[Chuck Lever]

On Jun 2, 2013, at 9:00 PM, Neil Brown <neilb@suse.de> wrote:

> As you probably know, since 3.7 (I think) Linux NFS has explicitly
> asked for machine credentials for certain requests rather than asking
> for root credentials as is previously did.
> This causes a regression for people who don't have any machine
> credentials configured and use "gssd -n".
> 
> I gather this was discussed on the mailing list earlier this year but
> not resolved.

It's resolved in 3.10-rc.

The kernel will attempt to use krb5i for lease management operations.  If that fails because there is no keytab available, it falls back to using AUTH_SYS.


> I would like to re-awaken the issue and offer a resolution (which has
> been tested and found effective by a customer).
> 
> Hence these three patches.  The first two are minor issues that I
> stumbled over while trying to understand the problem and are not
> critical but probably should be fixed.
> 
> The third addresses the above mentioned issue.  It introduces a
> variable "machine_uses_root_credentials" which is similar to the
> current "root_uses_machine_credentials".  It also adds a "-N" flag to
> set this variable.
> 
> I'm not certain what the defaults should be.  For backward
> compatibility it would be best if '-n' set the this new variable as
> well as clearing the old one, but then I'm not sure what exactly -N
> should do.
> 
> Comments welcome.
> 
> Thanks,
> NeilBrown
> 
> 
> 
> ---
> 
> Neil Brown (3):
>      krb5_utils: remove redundant array size.
>      krb5_util: don't give up on machine credential if hostname not available.
>      gssd: add -N option to use root credentials as machine credentials.
> 
> 
> utils/gssd/gssd.c      |    9 ++++++---
> utils/gssd/gssd.h      |    1 +
> utils/gssd/gssd.man    |   13 ++++++++++++-
> utils/gssd/gssd_proc.c |   12 +++++++-----
> utils/gssd/krb5_util.c |   10 +++++++---
> 5 files changed, 33 insertions(+), 12 deletions(-)
> 
> -- 
> Signature
> 

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: [PATCH 0/3] Various gssd fixes including machine-credential issue.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2532 bytes --]

On Sun, 2 Jun 2013 22:01:50 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:

> 
> On Jun 2, 2013, at 9:00 PM, Neil Brown <neilb@suse.de> wrote:
> 
> > As you probably know, since 3.7 (I think) Linux NFS has explicitly
> > asked for machine credentials for certain requests rather than asking
> > for root credentials as is previously did.
> > This causes a regression for people who don't have any machine
> > credentials configured and use "gssd -n".
> > 
> > I gather this was discussed on the mailing list earlier this year but
> > not resolved.
> 
> It's resolved in 3.10-rc.
> 
> The kernel will attempt to use krb5i for lease management operations.  If that fails because there is no keytab available, it falls back to using AUTH_SYS.

And if the server refuses to accept AUTH_SYS?

I guess this is commit 79d852bf5e7691dc7 ??  It seems to say that the server
should always accept AUTH_SYS ... is that right?

That commit isn't tagged for -stable.
So do we still need to make it work for 3.7,3.8,3.9 users?

Thanks,
NeilBrown

> 
> 
> > I would like to re-awaken the issue and offer a resolution (which has
> > been tested and found effective by a customer).
> > 
> > Hence these three patches.  The first two are minor issues that I
> > stumbled over while trying to understand the problem and are not
> > critical but probably should be fixed.
> > 
> > The third addresses the above mentioned issue.  It introduces a
> > variable "machine_uses_root_credentials" which is similar to the
> > current "root_uses_machine_credentials".  It also adds a "-N" flag to
> > set this variable.
> > 
> > I'm not certain what the defaults should be.  For backward
> > compatibility it would be best if '-n' set the this new variable as
> > well as clearing the old one, but then I'm not sure what exactly -N
> > should do.
> > 
> > Comments welcome.
> > 
> > Thanks,
> > NeilBrown
> > 
> > 
> > 
> > ---
> > 
> > Neil Brown (3):
> >      krb5_utils: remove redundant array size.
> >      krb5_util: don't give up on machine credential if hostname not available.
> >      gssd: add -N option to use root credentials as machine credentials.
> > 
> > 
> > utils/gssd/gssd.c      |    9 ++++++---
> > utils/gssd/gssd.h      |    1 +
> > utils/gssd/gssd.man    |   13 ++++++++++++-
> > utils/gssd/gssd_proc.c |   12 +++++++-----
> > utils/gssd/krb5_util.c |   10 +++++++---
> > 5 files changed, 33 insertions(+), 12 deletions(-)
> > 
> > -- 
> > Signature
> > 
> 


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [PATCH 0/3] Various gssd fixes including machine-credential issue.

[Chuck Lever]

On Jun 2, 2013, at 10:23 PM, NeilBrown <neilb@suse.de> wrote:

> On Sun, 2 Jun 2013 22:01:50 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:
> 
>> 
>> On Jun 2, 2013, at 9:00 PM, Neil Brown <neilb@suse.de> wrote:
>> 
>>> As you probably know, since 3.7 (I think) Linux NFS has explicitly
>>> asked for machine credentials for certain requests rather than asking
>>> for root credentials as is previously did.
>>> This causes a regression for people who don't have any machine
>>> credentials configured and use "gssd -n".
>>> 
>>> I gather this was discussed on the mailing list earlier this year but
>>> not resolved.
>> 
>> It's resolved in 3.10-rc.
>> 
>> The kernel will attempt to use krb5i for lease management operations.  If that fails because there is no keytab available, it falls back to using AUTH_SYS.
> 
> And if the server refuses to accept AUTH_SYS?
> 
> I guess this is commit 79d852bf5e7691dc7 ??

That's one of the subsequent bug fixes.  The initial change is commit 4edaa308.

> It seems to say that the server should always accept AUTH_SYS ... is that right?

If we ever find a server implementation that does not support either Kerberos or AUTH_SYS, we can add another step to the negotiation.

So far, despite RFC 3530 not requiring AUTH_SYS support on NFSv4 servers, I haven't found an implementation that does not support AUTH_SYS.  We have found one (FreeBSD) that does not support AUTH_NONE.  We do know that some servers allow administrators to control what security flavors are allowed for lease management.

> That commit isn't tagged for -stable.
> So do we still need to make it work for 3.7,3.8,3.9 users?

There are several commits that would need to be back-ported, starting with commit 4edaa308.  I am not certain they would apply cleanly to 3.[789], but a backport should not be difficult.

This change also requires that now gssd must be running on the client.  Otherwise without gssd a sec=sys mount hangs for a bit waiting for the upcall to time out (since the client will attempt to use krb5i for lease management operations).  Trond and Bruce have been discussing a change to address that.

> Thanks,
> NeilBrown
> 
>> 
>> 
>>> I would like to re-awaken the issue and offer a resolution (which has
>>> been tested and found effective by a customer).
>>> 
>>> Hence these three patches.  The first two are minor issues that I
>>> stumbled over while trying to understand the problem and are not
>>> critical but probably should be fixed.
>>> 
>>> The third addresses the above mentioned issue.  It introduces a
>>> variable "machine_uses_root_credentials" which is similar to the
>>> current "root_uses_machine_credentials".  It also adds a "-N" flag to
>>> set this variable.
>>> 
>>> I'm not certain what the defaults should be.  For backward
>>> compatibility it would be best if '-n' set the this new variable as
>>> well as clearing the old one, but then I'm not sure what exactly -N
>>> should do.
>>> 
>>> Comments welcome.
>>> 
>>> Thanks,
>>> NeilBrown
>>> 
>>> 
>>> 
>>> ---
>>> 
>>> Neil Brown (3):
>>>     krb5_utils: remove redundant array size.
>>>     krb5_util: don't give up on machine credential if hostname not available.
>>>     gssd: add -N option to use root credentials as machine credentials.
>>> 
>>> 
>>> utils/gssd/gssd.c      |    9 ++++++---
>>> utils/gssd/gssd.h      |    1 +
>>> utils/gssd/gssd.man    |   13 ++++++++++++-
>>> utils/gssd/gssd_proc.c |   12 +++++++-----
>>> utils/gssd/krb5_util.c |   10 +++++++---
>>> 5 files changed, 33 insertions(+), 12 deletions(-)
>>> 
>>> -- 
>>> Signature
>>> 
>> 
> 

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: [PATCH 0/3] Various gssd fixes including machine-credential issue.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2766 bytes --]

On Sun, 2 Jun 2013 22:45:16 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:

> 
> On Jun 2, 2013, at 10:23 PM, NeilBrown <neilb@suse.de> wrote:
> 
> > On Sun, 2 Jun 2013 22:01:50 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:
> > 
> >> 
> >> On Jun 2, 2013, at 9:00 PM, Neil Brown <neilb@suse.de> wrote:
> >> 
> >>> As you probably know, since 3.7 (I think) Linux NFS has explicitly
> >>> asked for machine credentials for certain requests rather than asking
> >>> for root credentials as is previously did.
> >>> This causes a regression for people who don't have any machine
> >>> credentials configured and use "gssd -n".
> >>> 
> >>> I gather this was discussed on the mailing list earlier this year but
> >>> not resolved.
> >> 
> >> It's resolved in 3.10-rc.
> >> 
> >> The kernel will attempt to use krb5i for lease management operations.  If that fails because there is no keytab available, it falls back to using AUTH_SYS.
> > 
> > And if the server refuses to accept AUTH_SYS?
> > 
> > I guess this is commit 79d852bf5e7691dc7 ??
> 
> That's one of the subsequent bug fixes.  The initial change is commit 4edaa308.
> 
> > It seems to say that the server should always accept AUTH_SYS ... is that right?
> 
> If we ever find a server implementation that does not support either Kerberos or AUTH_SYS, we can add another step to the negotiation.
> 
> So far, despite RFC 3530 not requiring AUTH_SYS support on NFSv4 servers, I haven't found an implementation that does not support AUTH_SYS.  We have found one (FreeBSD) that does not support AUTH_NONE.  We do know that some servers allow administrators to control what security flavors are allowed for lease management.
> 
> > That commit isn't tagged for -stable.
> > So do we still need to make it work for 3.7,3.8,3.9 users?
> 
> There are several commits that would need to be back-ported, starting with commit 4edaa308.  I am not certain they would apply cleanly to 3.[789], but a backport should not be difficult.
> 
> This change also requires that now gssd must be running on the client.  Otherwise without gssd a sec=sys mount hangs for a bit waiting for the upcall to time out (since the client will attempt to use krb5i for lease management operations).  Trond and Bruce have been discussing a change to address that.

Thanks for the explanation.  That all looks rather painful to back-port
though, especially as some of it isn't even written yet :-)
I think I'll stick with my "-N" option for openSUSE for now.

Do you think that supporting -N (or similar) so that the admin can ask for
root credentials to be used for SETCLIENTID requests is reasonable? i.e. what
do you think of my patch going in to nfs-utils anyway?

Thanks,
NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [PATCH 0/3] Various gssd fixes including machine-credential issue.

[Chuck Lever]

On Jun 2, 2013, at 11:01 PM, NeilBrown <neilb@suse.de> wrote:

> On Sun, 2 Jun 2013 22:45:16 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:
> 
>> 
>> On Jun 2, 2013, at 10:23 PM, NeilBrown <neilb@suse.de> wrote:
>> 
>>> On Sun, 2 Jun 2013 22:01:50 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:
>>> 
>>>> 
>>>> On Jun 2, 2013, at 9:00 PM, Neil Brown <neilb@suse.de> wrote:
>>>> 
>>>>> As you probably know, since 3.7 (I think) Linux NFS has explicitly
>>>>> asked for machine credentials for certain requests rather than asking
>>>>> for root credentials as is previously did.
>>>>> This causes a regression for people who don't have any machine
>>>>> credentials configured and use "gssd -n".
>>>>> 
>>>>> I gather this was discussed on the mailing list earlier this year but
>>>>> not resolved.
>>>> 
>>>> It's resolved in 3.10-rc.
>>>> 
>>>> The kernel will attempt to use krb5i for lease management operations.  If that fails because there is no keytab available, it falls back to using AUTH_SYS.
>>> 
>>> And if the server refuses to accept AUTH_SYS?
>>> 
>>> I guess this is commit 79d852bf5e7691dc7 ??
>> 
>> That's one of the subsequent bug fixes.  The initial change is commit 4edaa308.
>> 
>>> It seems to say that the server should always accept AUTH_SYS ... is that right?
>> 
>> If we ever find a server implementation that does not support either Kerberos or AUTH_SYS, we can add another step to the negotiation.
>> 
>> So far, despite RFC 3530 not requiring AUTH_SYS support on NFSv4 servers, I haven't found an implementation that does not support AUTH_SYS.  We have found one (FreeBSD) that does not support AUTH_NONE.  We do know that some servers allow administrators to control what security flavors are allowed for lease management.
>> 
>>> That commit isn't tagged for -stable.
>>> So do we still need to make it work for 3.7,3.8,3.9 users?
>> 
>> There are several commits that would need to be back-ported, starting with commit 4edaa308.  I am not certain they would apply cleanly to 3.[789], but a backport should not be difficult.
>> 
>> This change also requires that now gssd must be running on the client.  Otherwise without gssd a sec=sys mount hangs for a bit waiting for the upcall to time out (since the client will attempt to use krb5i for lease management operations).  Trond and Bruce have been discussing a change to address that.
> 
> Thanks for the explanation.  That all looks rather painful to back-port
> though, especially as some of it isn't even written yet :-)
> I think I'll stick with my "-N" option for openSUSE for now.
> 
> Do you think that supporting -N (or similar) so that the admin can ask for
> root credentials to be used for SETCLIENTID requests is reasonable? i.e. what
> do you think of my patch going in to nfs-utils anyway?

So, let me confirm my understanding of your proposed changes: a user logs in as root, then kinit's with her own user principal.  That establishes a Kerberos credential for root to use, and "-N" makes gssd return this credential when the kernel requests "service=*".

Lease management is shared among all NFS users on a client.  In particular, machine credentials give us two important features that cannot be matched by user credentials:

  o  Machine credentials never expire even when users log out.  On a
     multi-user client, you don't want credential expiry or a user
     logout to cause lease management to stop working for other users. 

  o  Machine credentials are always the same no matter who is logged
     in.  The client has to use the same credential every time for
     lease management, or else servers return CLID_INUSE and prevent a
     fresh lease from being established (at least until the existing
     lease for that clientid expires on the server).

Using root's credentials for lease management makes it likely one of these two bullets will end up in your foot.  Specifically for the use case where one user is always using the same client (say, a student's laptop), "-N" might work fine.  For other use cases, like a shared build machine that doesn't have a keytab, "-N" would not work reliably.

With 3.10, if we can't use a Kerberos host principal for lease management, we'll use a less secure equivalent.  The point of backing off the security flavor for lease management is that AUTH_SYS with UID 0, or even AUTH_NONE, are essentially machine credentials.  They don't expire, and the client will present a consistent credential for lease management, no matter what user logs in or leaves the machine.

So, yes, machine credentials are a pain in the ass, and have been for years.  I'm sorry this took 3 kernel releases to figure out, but I'm hoping that the path we're on with these kernel changes is towards more transparent support for a wider array of use cases.  NFSv4.1 also offers some relief in this area (see SP4_MACH_CRED, which I think we do not support quite yet).

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: [PATCH 0/3] Various gssd fixes including machine-credential issue.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 6873 bytes --]

On Mon, 3 Jun 2013 00:32:54 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:

> 
> On Jun 2, 2013, at 11:01 PM, NeilBrown <neilb@suse.de> wrote:
> 
> > On Sun, 2 Jun 2013 22:45:16 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:
> > 
> >> 
> >> On Jun 2, 2013, at 10:23 PM, NeilBrown <neilb@suse.de> wrote:
> >> 
> >>> On Sun, 2 Jun 2013 22:01:50 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:
> >>> 
> >>>> 
> >>>> On Jun 2, 2013, at 9:00 PM, Neil Brown <neilb@suse.de> wrote:
> >>>> 
> >>>>> As you probably know, since 3.7 (I think) Linux NFS has explicitly
> >>>>> asked for machine credentials for certain requests rather than asking
> >>>>> for root credentials as is previously did.
> >>>>> This causes a regression for people who don't have any machine
> >>>>> credentials configured and use "gssd -n".
> >>>>> 
> >>>>> I gather this was discussed on the mailing list earlier this year but
> >>>>> not resolved.
> >>>> 
> >>>> It's resolved in 3.10-rc.
> >>>> 
> >>>> The kernel will attempt to use krb5i for lease management operations.  If that fails because there is no keytab available, it falls back to using AUTH_SYS.
> >>> 
> >>> And if the server refuses to accept AUTH_SYS?
> >>> 
> >>> I guess this is commit 79d852bf5e7691dc7 ??
> >> 
> >> That's one of the subsequent bug fixes.  The initial change is commit 4edaa308.
> >> 
> >>> It seems to say that the server should always accept AUTH_SYS ... is that right?
> >> 
> >> If we ever find a server implementation that does not support either Kerberos or AUTH_SYS, we can add another step to the negotiation.
> >> 
> >> So far, despite RFC 3530 not requiring AUTH_SYS support on NFSv4 servers, I haven't found an implementation that does not support AUTH_SYS.  We have found one (FreeBSD) that does not support AUTH_NONE.  We do know that some servers allow administrators to control what security flavors are allowed for lease management.
> >> 
> >>> That commit isn't tagged for -stable.
> >>> So do we still need to make it work for 3.7,3.8,3.9 users?
> >> 
> >> There are several commits that would need to be back-ported, starting with commit 4edaa308.  I am not certain they would apply cleanly to 3.[789], but a backport should not be difficult.
> >> 
> >> This change also requires that now gssd must be running on the client.  Otherwise without gssd a sec=sys mount hangs for a bit waiting for the upcall to time out (since the client will attempt to use krb5i for lease management operations).  Trond and Bruce have been discussing a change to address that.
> > 
> > Thanks for the explanation.  That all looks rather painful to back-port
> > though, especially as some of it isn't even written yet :-)
> > I think I'll stick with my "-N" option for openSUSE for now.
> > 
> > Do you think that supporting -N (or similar) so that the admin can ask for
> > root credentials to be used for SETCLIENTID requests is reasonable? i.e. what
> > do you think of my patch going in to nfs-utils anyway?
> 
> So, let me confirm my understanding of your proposed changes: a user logs in as root, then kinit's with her own user principal.  That establishes a Kerberos credential for root to use, and "-N" makes gssd return this credential when the kernel requests "service=*".

Correct.

> 
> Lease management is shared among all NFS users on a client.  In particular, machine credentials give us two important features that cannot be matched by user credentials:
> 
>   o  Machine credentials never expire even when users log out.  On a
>      multi-user client, you don't want credential expiry or a user
>      logout to cause lease management to stop working for other users. 

... while on a single-user client, this isn't really an issue.

And I can easily imagine a server admin not wanting to hand out credentials
that never expire, but still requiring strong credentials for lease
management.

> 
>   o  Machine credentials are always the same no matter who is logged
>      in.  The client has to use the same credential every time for
>      lease management, or else servers return CLID_INUSE and prevent a
>      fresh lease from being established (at least until the existing
>      lease for that clientid expires on the server).

Again, not really an issue for a single-user client.  And single-user
machines are certainly a common use-case these days.

> 
> Using root's credentials for lease management makes it likely one of these two bullets will end up in your foot.  Specifically for the use case where one user is always using the same client (say, a student's laptop), "-N" might work fine.  For other use cases, like a shared build machine that doesn't have a keytab, "-N" would not work reliably.

Completely agree.  So -N should certainly not be the default.

> 
> With 3.10, if we can't use a Kerberos host principal for lease management, we'll use a less secure equivalent.  The point of backing off the security flavor for lease management is that AUTH_SYS with UID 0, or even AUTH_NONE, are essentially machine credentials.  They don't expire, and the client will present a consistent credential for lease management, no matter what user logs in or leaves the machine.
> 
> So, yes, machine credentials are a pain in the ass, and have been for years.  I'm sorry this took 3 kernel releases to figure out, but I'm hoping that the path we're on with these kernel changes is towards more transparent support for a wider array of use cases.  NFSv4.1 also offers some relief in this area (see SP4_MACH_CRED, which I think we do not support quite yet).

All sounds very good, but doesn't seem to address the issue.

For the openSUSE user who raised this, he simply does not have a machine
credential. I don't know why, maybe the server admin doesn't want to hand
them out.  But this hasn't been a problem because "gssd -n" meant that a
machine credential would never be used.
But now,  with 3.[789], "gssd -n" still wants the machine credential that
doesn't exist.

It is possible that with 3.10, he might get away with doing lease management
with AUTH_SYS, but that seems unlikely.  Given that an integrity protecting
security flavour is now recommended for SETCLIENTID, his server admin
could easily make a case that AUTH_SYS won't be accepted.  In that case he
would still have a problem.

Given that "gssd -n" has meant "no machine credential needed" in the past, I
think it is important to preserve the possibility of running secure NFS with
no machine credential.  I don't see how your changes (valuable though they
are) achieve that.

So I think we either need to change "gssd -n" to mean "never use machine
credentials" or add something like "gssd -N", so the two different possible
uses of machine credentials can be separately controlled.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [PATCH 0/3] Various gssd fixes including machine-credential issue.

[Chuck Lever]

On Jun 3, 2013, at 7:30 PM, NeilBrown <neilb@suse.de> wrote:

> On Mon, 3 Jun 2013 00:32:54 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:
> 
>> 
>> On Jun 2, 2013, at 11:01 PM, NeilBrown <neilb@suse.de> wrote:
>> 
>>> On Sun, 2 Jun 2013 22:45:16 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:
>>> 
>>>> 
>>>> On Jun 2, 2013, at 10:23 PM, NeilBrown <neilb@suse.de> wrote:
>>>> 
>>>>> On Sun, 2 Jun 2013 22:01:50 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:
>>>>> 
>>>>>> 
>>>>>> On Jun 2, 2013, at 9:00 PM, Neil Brown <neilb@suse.de> wrote:
>>>>>> 
>>>>>>> As you probably know, since 3.7 (I think) Linux NFS has explicitly
>>>>>>> asked for machine credentials for certain requests rather than asking
>>>>>>> for root credentials as is previously did.
>>>>>>> This causes a regression for people who don't have any machine
>>>>>>> credentials configured and use "gssd -n".
>>>>>>> 
>>>>>>> I gather this was discussed on the mailing list earlier this year but
>>>>>>> not resolved.
>>>>>> 
>>>>>> It's resolved in 3.10-rc.
>>>>>> 
>>>>>> The kernel will attempt to use krb5i for lease management operations.  If that fails because there is no keytab available, it falls back to using AUTH_SYS.
>>>>> 
>>>>> And if the server refuses to accept AUTH_SYS?
>>>>> 
>>>>> I guess this is commit 79d852bf5e7691dc7 ??
>>>> 
>>>> That's one of the subsequent bug fixes.  The initial change is commit 4edaa308.
>>>> 
>>>>> It seems to say that the server should always accept AUTH_SYS ... is that right?
>>>> 
>>>> If we ever find a server implementation that does not support either Kerberos or AUTH_SYS, we can add another step to the negotiation.
>>>> 
>>>> So far, despite RFC 3530 not requiring AUTH_SYS support on NFSv4 servers, I haven't found an implementation that does not support AUTH_SYS.  We have found one (FreeBSD) that does not support AUTH_NONE.  We do know that some servers allow administrators to control what security flavors are allowed for lease management.
>>>> 
>>>>> That commit isn't tagged for -stable.
>>>>> So do we still need to make it work for 3.7,3.8,3.9 users?
>>>> 
>>>> There are several commits that would need to be back-ported, starting with commit 4edaa308.  I am not certain they would apply cleanly to 3.[789], but a backport should not be difficult.
>>>> 
>>>> This change also requires that now gssd must be running on the client.  Otherwise without gssd a sec=sys mount hangs for a bit waiting for the upcall to time out (since the client will attempt to use krb5i for lease management operations).  Trond and Bruce have been discussing a change to address that.
>>> 
>>> Thanks for the explanation.  That all looks rather painful to back-port
>>> though, especially as some of it isn't even written yet :-)
>>> I think I'll stick with my "-N" option for openSUSE for now.
>>> 
>>> Do you think that supporting -N (or similar) so that the admin can ask for
>>> root credentials to be used for SETCLIENTID requests is reasonable? i.e. what
>>> do you think of my patch going in to nfs-utils anyway?
>> 
>> So, let me confirm my understanding of your proposed changes: a user logs in as root, then kinit's with her own user principal.  That establishes a Kerberos credential for root to use, and "-N" makes gssd return this credential when the kernel requests "service=*".
> 
> Correct.
> 
>> 
>> Lease management is shared among all NFS users on a client.  In particular, machine credentials give us two important features that cannot be matched by user credentials:
>> 
>>  o  Machine credentials never expire even when users log out.  On a
>>     multi-user client, you don't want credential expiry or a user
>>     logout to cause lease management to stop working for other users. 
> 
> ... while on a single-user client, this isn't really an issue.
> 
> And I can easily imagine a server admin not wanting to hand out credentials
> that never expire, but still requiring strong credentials for lease
> management.

Why, exactly, would strong credentials be required for lease management, but an admin would choose not to hand out keytabs?

> 
>> 
>>  o  Machine credentials are always the same no matter who is logged
>>     in.  The client has to use the same credential every time for
>>     lease management, or else servers return CLID_INUSE and prevent a
>>     fresh lease from being established (at least until the existing
>>     lease for that clientid expires on the server).
> 
> Again, not really an issue for a single-user client.  And single-user
> machines are certainly a common use-case these days.
> 
>> 
>> Using root's credentials for lease management makes it likely one of these two bullets will end up in your foot.  Specifically for the use case where one user is always using the same client (say, a student's laptop), "-N" might work fine.  For other use cases, like a shared build machine that doesn't have a keytab, "-N" would not work reliably.
> 
> Completely agree.  So -N should certainly not be the default.
> 
>> 
>> With 3.10, if we can't use a Kerberos host principal for lease management, we'll use a less secure equivalent.  The point of backing off the security flavor for lease management is that AUTH_SYS with UID 0, or even AUTH_NONE, are essentially machine credentials.  They don't expire, and the client will present a consistent credential for lease management, no matter what user logs in or leaves the machine.
>> 
>> So, yes, machine credentials are a pain in the ass, and have been for years.  I'm sorry this took 3 kernel releases to figure out, but I'm hoping that the path we're on with these kernel changes is towards more transparent support for a wider array of use cases.  NFSv4.1 also offers some relief in this area (see SP4_MACH_CRED, which I think we do not support quite yet).
> 
> All sounds very good, but doesn't seem to address the issue.

> For the openSUSE user who raised this, he simply does not have a machine
> credential. I don't know why, maybe the server admin doesn't want to hand
> them out.

Handing out keytabs for every NFS client that wants to use Kerberized NFS is onerous for some administrators.

> But this hasn't been a problem because "gssd -n" meant that a
> machine credential would never be used.

No, it meant that root's user credential was used instead of a service credential for lease  management.  As I explained above, that solution has some compromises for some important use cases.

> But now,  with 3.[789], "gssd -n" still wants the machine credential that
> doesn't exist.

The change that introduced the requirement for a machine credential is commit 68c97153 "SUNRPC: Clean up the RPCSEC_GSS service ticket requests", Tue Jan 3 13:22:46 2012, which was merged long before 3.7.

The Uniform Client String changes in 3.7 merely made the machine credential requirement more obtrusive.  The new broken "-n" behavior is an unintended side effect.

> It is possible that with 3.10, he might get away with doing lease management
> with AUTH_SYS, but that seems unlikely.  Given that an integrity protecting
> security flavour is now recommended for SETCLIENTID, his server admin
> could easily make a case that AUTH_SYS won't be accepted.  In that case he
> would still have a problem.

His problem then would be with his server admin, not with us.

If a server admin wants to protect his server by denying the use of AUTH_SYS for lease management, he has every right to do that.  But he must also therefore understand the position that puts his users in.

> Given that "gssd -n" has meant "no machine credential needed" in the past,

It never meant that.  It meant "use root's credential as the machine credential".  I think we should be clear about exactly what "-n" is.

I also believe "-n" (or "-N") is a workaround.  Logging in as root and authenticating is a pain for users, who have already logged in once.

> I think it is important to preserve the possibility of running secure NFS with
> no machine credential.  I don't see how your changes (valuable though they are)
> achieve that.

Please explain how allowing lease management to occur using AUTH_SYS is somehow a major new exposure.  A mixture of sec=sys and sec=krb5 mounts today means there's a good chance that a sec=sys mount might occur first, and thus lease management uses AUTH_SYS anyway.

Can you share your attack model?  Why do you believe that using AUTH_SYS for lease management on sec=krb5 mounts will be unacceptable to users?  I suspect that most will not notice or care.

> So I think we either need to change "gssd -n" to mean "never use machine
> credentials" or add something like "gssd -N", so the two different possible
> uses of machine credentials can be separately controlled.

I'm not convinced that "-N" is the right way to fix this.  Another knob to turn means more testing for us, more documentation to write and maintain, and so on.  It doesn't get us any farther forward.  The difference between "-n" (which no longer works) and "-N" is subtle and confusing.  Explaining when to use "-N" versus "-n" is not a job I would like to do, and I'm not interested in taking the support calls either.  :-p

If you think "-n" is truly valuable (I have my doubts), I can go back and try to make that work again.  Maybe we should first revisit why UCS caused "gssd -n" to stop working and start again from there.

You should keep in mind that I'm just the code monkey here.  As I said above, the machine credential policy was set long before 3.7.  You'll need to take up with Trond whether we want to conveniently enable the use of user credentials for lease management.  My understanding is we comply with the standard in this regard.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: [PATCH 0/3] Various gssd fixes including machine-credential issue.

[Chuck Lever]

On Jun 3, 2013, at 9:13 PM, Chuck Lever <chuck.lever@oracle.com> wrote:

> 
> On Jun 3, 2013, at 7:30 PM, NeilBrown <neilb@suse.de> wrote:
> 
>> On Mon, 3 Jun 2013 00:32:54 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:
>> 
>>> 
>>> On Jun 2, 2013, at 11:01 PM, NeilBrown <neilb@suse.de> wrote:
>>> 
>>>> On Sun, 2 Jun 2013 22:45:16 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:
>>>> 
>>>>> 
>>>>> On Jun 2, 2013, at 10:23 PM, NeilBrown <neilb@suse.de> wrote:
>>>>> 
>>>>>> On Sun, 2 Jun 2013 22:01:50 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:
>>>>>> 
>>>>>>> 
>>>>>>> On Jun 2, 2013, at 9:00 PM, Neil Brown <neilb@suse.de> wrote:
>>>>>>> 
>>>>>>>> As you probably know, since 3.7 (I think) Linux NFS has explicitly
>>>>>>>> asked for machine credentials for certain requests rather than asking
>>>>>>>> for root credentials as is previously did.
>>>>>>>> This causes a regression for people who don't have any machine
>>>>>>>> credentials configured and use "gssd -n".
>>>>>>>> 
>>>>>>>> I gather this was discussed on the mailing list earlier this year but
>>>>>>>> not resolved.
>>>>>>> 
>>>>>>> It's resolved in 3.10-rc.
>>>>>>> 
>>>>>>> The kernel will attempt to use krb5i for lease management operations.  If that fails because there is no keytab available, it falls back to using AUTH_SYS.
>>>>>> 
>>>>>> And if the server refuses to accept AUTH_SYS?
>>>>>> 
>>>>>> I guess this is commit 79d852bf5e7691dc7 ??
>>>>> 
>>>>> That's one of the subsequent bug fixes.  The initial change is commit 4edaa308.
>>>>> 
>>>>>> It seems to say that the server should always accept AUTH_SYS ... is that right?
>>>>> 
>>>>> If we ever find a server implementation that does not support either Kerberos or AUTH_SYS, we can add another step to the negotiation.
>>>>> 
>>>>> So far, despite RFC 3530 not requiring AUTH_SYS support on NFSv4 servers, I haven't found an implementation that does not support AUTH_SYS.  We have found one (FreeBSD) that does not support AUTH_NONE.  We do know that some servers allow administrators to control what security flavors are allowed for lease management.
>>>>> 
>>>>>> That commit isn't tagged for -stable.
>>>>>> So do we still need to make it work for 3.7,3.8,3.9 users?
>>>>> 
>>>>> There are several commits that would need to be back-ported, starting with commit 4edaa308.  I am not certain they would apply cleanly to 3.[789], but a backport should not be difficult.
>>>>> 
>>>>> This change also requires that now gssd must be running on the client.  Otherwise without gssd a sec=sys mount hangs for a bit waiting for the upcall to time out (since the client will attempt to use krb5i for lease management operations).  Trond and Bruce have been discussing a change to address that.
>>>> 
>>>> Thanks for the explanation.  That all looks rather painful to back-port
>>>> though, especially as some of it isn't even written yet :-)
>>>> I think I'll stick with my "-N" option for openSUSE for now.
>>>> 
>>>> Do you think that supporting -N (or similar) so that the admin can ask for
>>>> root credentials to be used for SETCLIENTID requests is reasonable? i.e. what
>>>> do you think of my patch going in to nfs-utils anyway?
>>> 
>>> So, let me confirm my understanding of your proposed changes: a user logs in as root, then kinit's with her own user principal.  That establishes a Kerberos credential for root to use, and "-N" makes gssd return this credential when the kernel requests "service=*".
>> 
>> Correct.
>> 
>>> 
>>> Lease management is shared among all NFS users on a client.  In particular, machine credentials give us two important features that cannot be matched by user credentials:
>>> 
>>> o  Machine credentials never expire even when users log out.  On a
>>>    multi-user client, you don't want credential expiry or a user
>>>    logout to cause lease management to stop working for other users. 
>> 
>> ... while on a single-user client, this isn't really an issue.
>> 
>> And I can easily imagine a server admin not wanting to hand out credentials
>> that never expire, but still requiring strong credentials for lease
>> management.
> 
> Why, exactly, would strong credentials be required for lease management, but an admin would choose not to hand out keytabs?
> 
>> 
>>> 
>>> o  Machine credentials are always the same no matter who is logged
>>>    in.  The client has to use the same credential every time for
>>>    lease management, or else servers return CLID_INUSE and prevent a
>>>    fresh lease from being established (at least until the existing
>>>    lease for that clientid expires on the server).
>> 
>> Again, not really an issue for a single-user client.  And single-user
>> machines are certainly a common use-case these days.
>> 
>>> 
>>> Using root's credentials for lease management makes it likely one of these two bullets will end up in your foot.  Specifically for the use case where one user is always using the same client (say, a student's laptop), "-N" might work fine.  For other use cases, like a shared build machine that doesn't have a keytab, "-N" would not work reliably.
>> 
>> Completely agree.  So -N should certainly not be the default.
>> 
>>> 
>>> With 3.10, if we can't use a Kerberos host principal for lease management, we'll use a less secure equivalent.  The point of backing off the security flavor for lease management is that AUTH_SYS with UID 0, or even AUTH_NONE, are essentially machine credentials.  They don't expire, and the client will present a consistent credential for lease management, no matter what user logs in or leaves the machine.
>>> 
>>> So, yes, machine credentials are a pain in the ass, and have been for years.  I'm sorry this took 3 kernel releases to figure out, but I'm hoping that the path we're on with these kernel changes is towards more transparent support for a wider array of use cases.  NFSv4.1 also offers some relief in this area (see SP4_MACH_CRED, which I think we do not support quite yet).
>> 
>> All sounds very good, but doesn't seem to address the issue.
> 
>> For the openSUSE user who raised this, he simply does not have a machine
>> credential. I don't know why, maybe the server admin doesn't want to hand
>> them out.
> 
> Handing out keytabs for every NFS client that wants to use Kerberized NFS is onerous for some administrators.
> 
>> But this hasn't been a problem because "gssd -n" meant that a
>> machine credential would never be used.
> 
> No, it meant that root's user credential was used instead of a service credential for lease  management.  As I explained above, that solution has some compromises for some important use cases.

Correction:

rpc.gssd(8) on Fedora 18 says:

> With the -n option, "machine credentials" will not be used for accesses by UID 0.


In other words, normally root doesn't have a Kerberos ticket.  The kernel uses the machine's credentials for UID 0.  "-n" means only that root has to kinit.


>> But now,  with 3.[789], "gssd -n" still wants the machine credential that
>> doesn't exist.
> 
> The change that introduced the requirement for a machine credential is commit 68c97153 "SUNRPC: Clean up the RPCSEC_GSS service ticket requests", Tue Jan 3 13:22:46 2012, which was merged long before 3.7.
> 
> The Uniform Client String changes in 3.7 merely made the machine credential requirement more obtrusive.  The new broken "-n" behavior is an unintended side effect.

I was wondering what exactly about 3.7's UCS changes exposed the requirement for a machine credential, so I built a 3.6 kernel and tried it.

The key difference between 3.6 and 3.7 is that 3.6 performs the SETCLIENTID much later, after a user has logged in and kinit'd, and is actually doing an OPEN.

3.6 requests a machine credential, but if that fails, it asks for the OPEN user's credential for SETCLIENTID.  The retried upcall request is for "service=<null> UID=nnnn".

If root kinit's for the mount, but the user does not kinit before accessing the mount, the SETCLIENTID still fails!  Root's ticket and "gssd -n" has nothing to do with SETCLIENTID's security flavor, as far as I can tell.

With 3.7, the SETCLIENTID is done right at mount time.  There is no user with credentials trying to do an OPEN, so the only credential the kernel tries is the machine's credential.  Perhaps the kernel could retry the upcall with "server=<null> UID=0".

>> It is possible that with 3.10, he might get away with doing lease management
>> with AUTH_SYS, but that seems unlikely.  Given that an integrity protecting
>> security flavour is now recommended for SETCLIENTID, his server admin
>> could easily make a case that AUTH_SYS won't be accepted.  In that case he
>> would still have a problem.
> 
> His problem then would be with his server admin, not with us.
> 
> If a server admin wants to protect his server by denying the use of AUTH_SYS for lease management, he has every right to do that.  But he must also therefore understand the position that puts his users in.
> 
>> Given that "gssd -n" has meant "no machine credential needed" in the past,
> 
> It never meant that.  It meant "use root's credential as the machine credential".  I think we should be clear about exactly what "-n" is.

Right, I should heed my own words.

> I also believe "-n" (or "-N") is a workaround.  Logging in as root and authenticating is a pain for users, who have already logged in once.

This is still true, however.

>> I think it is important to preserve the possibility of running secure NFS with
>> no machine credential.  I don't see how your changes (valuable though they are)
>> achieve that.
> 
> Please explain how allowing lease management to occur using AUTH_SYS is somehow a major new exposure.  A mixture of sec=sys and sec=krb5 mounts today means there's a good chance that a sec=sys mount might occur first, and thus lease management uses AUTH_SYS anyway.
> 
> Can you share your attack model?  Why do you believe that using AUTH_SYS for lease management on sec=krb5 mounts will be unacceptable to users?  I suspect that most will not notice or care.

I'm still interested in knowing if we have a palpable hole in the current solution in 3.10.

>> So I think we either need to change "gssd -n" to mean "never use machine
>> credentials" or add something like "gssd -N", so the two different possible
>> uses of machine credentials can be separately controlled.
> 
> I'm not convinced that "-N" is the right way to fix this.  Another knob to turn means more testing for us, more documentation to write and maintain, and so on.  It doesn't get us any farther forward.  The difference between "-n" (which no longer works) and "-N" is subtle and confusing.  Explaining when to use "-N" versus "-n" is not a job I would like to do, and I'm not interested in taking the support calls either.  :-p

Let me restate this without all the cleverness.

Trond's recommendation was to tell gssd exactly where to find the machine credential if it's not in a keytab: basically an option that says "look in this file for that user credential".  That was our first try, but it was clumsy.

You want "gssd -N" to mean specifically "use root's ticket as the machine credential."  Then "-n -N" would mean "force root to kinit, and use root's ticket as the machine credential."  (Perhaps "-N" doesn't make sense unless "-n" is also specified...?)

But that's still not the same as 3.6's behavior.  And, it would still be a regression, as I understand it, because a user space change would be needed after a kernel upgrade.  For that reason, for 3.[789] we probably want to find a kernel-only solution.

Command line options on gssd are really poor from a usability standpoint.  IMO in the long run we want "mount sec=krb5 without a client keytab" just to work.  No monkeying with gssd.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: [PATCH 0/3] Various gssd fixes including machine-credential issue.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 16018 bytes --]

On Tue, 4 Jun 2013 15:16:01 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:

> 
> On Jun 3, 2013, at 9:13 PM, Chuck Lever <chuck.lever@oracle.com> wrote:
> 
> > 
> > On Jun 3, 2013, at 7:30 PM, NeilBrown <neilb@suse.de> wrote:
> > 
> >> On Mon, 3 Jun 2013 00:32:54 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:
> >> 
> >>> 
> >>> On Jun 2, 2013, at 11:01 PM, NeilBrown <neilb@suse.de> wrote:
> >>> 
> >>>> On Sun, 2 Jun 2013 22:45:16 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:
> >>>> 
> >>>>> 
> >>>>> On Jun 2, 2013, at 10:23 PM, NeilBrown <neilb@suse.de> wrote:
> >>>>> 
> >>>>>> On Sun, 2 Jun 2013 22:01:50 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:
> >>>>>> 
> >>>>>>> 
> >>>>>>> On Jun 2, 2013, at 9:00 PM, Neil Brown <neilb@suse.de> wrote:
> >>>>>>> 
> >>>>>>>> As you probably know, since 3.7 (I think) Linux NFS has explicitly
> >>>>>>>> asked for machine credentials for certain requests rather than asking
> >>>>>>>> for root credentials as is previously did.
> >>>>>>>> This causes a regression for people who don't have any machine
> >>>>>>>> credentials configured and use "gssd -n".
> >>>>>>>> 
> >>>>>>>> I gather this was discussed on the mailing list earlier this year but
> >>>>>>>> not resolved.
> >>>>>>> 
> >>>>>>> It's resolved in 3.10-rc.
> >>>>>>> 
> >>>>>>> The kernel will attempt to use krb5i for lease management operations.  If that fails because there is no keytab available, it falls back to using AUTH_SYS.
> >>>>>> 
> >>>>>> And if the server refuses to accept AUTH_SYS?
> >>>>>> 
> >>>>>> I guess this is commit 79d852bf5e7691dc7 ??
> >>>>> 
> >>>>> That's one of the subsequent bug fixes.  The initial change is commit 4edaa308.
> >>>>> 
> >>>>>> It seems to say that the server should always accept AUTH_SYS ... is that right?
> >>>>> 
> >>>>> If we ever find a server implementation that does not support either Kerberos or AUTH_SYS, we can add another step to the negotiation.
> >>>>> 
> >>>>> So far, despite RFC 3530 not requiring AUTH_SYS support on NFSv4 servers, I haven't found an implementation that does not support AUTH_SYS.  We have found one (FreeBSD) that does not support AUTH_NONE.  We do know that some servers allow administrators to control what security flavors are allowed for lease management.
> >>>>> 
> >>>>>> That commit isn't tagged for -stable.
> >>>>>> So do we still need to make it work for 3.7,3.8,3.9 users?
> >>>>> 
> >>>>> There are several commits that would need to be back-ported, starting with commit 4edaa308.  I am not certain they would apply cleanly to 3.[789], but a backport should not be difficult.
> >>>>> 
> >>>>> This change also requires that now gssd must be running on the client.  Otherwise without gssd a sec=sys mount hangs for a bit waiting for the upcall to time out (since the client will attempt to use krb5i for lease management operations).  Trond and Bruce have been discussing a change to address that.
> >>>> 
> >>>> Thanks for the explanation.  That all looks rather painful to back-port
> >>>> though, especially as some of it isn't even written yet :-)
> >>>> I think I'll stick with my "-N" option for openSUSE for now.
> >>>> 
> >>>> Do you think that supporting -N (or similar) so that the admin can ask for
> >>>> root credentials to be used for SETCLIENTID requests is reasonable? i.e. what
> >>>> do you think of my patch going in to nfs-utils anyway?
> >>> 
> >>> So, let me confirm my understanding of your proposed changes: a user logs in as root, then kinit's with her own user principal.  That establishes a Kerberos credential for root to use, and "-N" makes gssd return this credential when the kernel requests "service=*".
> >> 
> >> Correct.
> >> 
> >>> 
> >>> Lease management is shared among all NFS users on a client.  In particular, machine credentials give us two important features that cannot be matched by user credentials:
> >>> 
> >>> o  Machine credentials never expire even when users log out.  On a
> >>>    multi-user client, you don't want credential expiry or a user
> >>>    logout to cause lease management to stop working for other users. 
> >> 
> >> ... while on a single-user client, this isn't really an issue.
> >> 
> >> And I can easily imagine a server admin not wanting to hand out credentials
> >> that never expire, but still requiring strong credentials for lease
> >> management.
> > 
> > Why, exactly, would strong credentials be required for lease management, but an admin would choose not to hand out keytabs?
> > 
> >> 
> >>> 
> >>> o  Machine credentials are always the same no matter who is logged
> >>>    in.  The client has to use the same credential every time for
> >>>    lease management, or else servers return CLID_INUSE and prevent a
> >>>    fresh lease from being established (at least until the existing
> >>>    lease for that clientid expires on the server).
> >> 
> >> Again, not really an issue for a single-user client.  And single-user
> >> machines are certainly a common use-case these days.
> >> 
> >>> 
> >>> Using root's credentials for lease management makes it likely one of these two bullets will end up in your foot.  Specifically for the use case where one user is always using the same client (say, a student's laptop), "-N" might work fine.  For other use cases, like a shared build machine that doesn't have a keytab, "-N" would not work reliably.
> >> 
> >> Completely agree.  So -N should certainly not be the default.
> >> 
> >>> 
> >>> With 3.10, if we can't use a Kerberos host principal for lease management, we'll use a less secure equivalent.  The point of backing off the security flavor for lease management is that AUTH_SYS with UID 0, or even AUTH_NONE, are essentially machine credentials.  They don't expire, and the client will present a consistent credential for lease management, no matter what user logs in or leaves the machine.
> >>> 
> >>> So, yes, machine credentials are a pain in the ass, and have been for years.  I'm sorry this took 3 kernel releases to figure out, but I'm hoping that the path we're on with these kernel changes is towards more transparent support for a wider array of use cases.  NFSv4.1 also offers some relief in this area (see SP4_MACH_CRED, which I think we do not support quite yet).
> >> 
> >> All sounds very good, but doesn't seem to address the issue.
> > 
> >> For the openSUSE user who raised this, he simply does not have a machine
> >> credential. I don't know why, maybe the server admin doesn't want to hand
> >> them out.
> > 
> > Handing out keytabs for every NFS client that wants to use Kerberized NFS is onerous for some administrators.
> > 
> >> But this hasn't been a problem because "gssd -n" meant that a
> >> machine credential would never be used.
> > 
> > No, it meant that root's user credential was used instead of a service credential for lease  management.  As I explained above, that solution has some compromises for some important use cases.
> 
> Correction:
> 
> rpc.gssd(8) on Fedora 18 says:
> 
> > With the -n option, "machine credentials" will not be used for accesses by UID 0.
> 
> 
> In other words, normally root doesn't have a Kerberos ticket.  The kernel uses the machine's credentials for UID 0.  "-n" means only that root has to kinit.
> 
> 
> >> But now,  with 3.[789], "gssd -n" still wants the machine credential that
> >> doesn't exist.
> > 
> > The change that introduced the requirement for a machine credential is commit 68c97153 "SUNRPC: Clean up the RPCSEC_GSS service ticket requests", Tue Jan 3 13:22:46 2012, which was merged long before 3.7.
> > 
> > The Uniform Client String changes in 3.7 merely made the machine credential requirement more obtrusive.  The new broken "-n" behavior is an unintended side effect.
> 
> I was wondering what exactly about 3.7's UCS changes exposed the requirement for a machine credential, so I built a 3.6 kernel and tried it.
> 
> The key difference between 3.6 and 3.7 is that 3.6 performs the SETCLIENTID much later, after a user has logged in and kinit'd, and is actually doing an OPEN.
> 
> 3.6 requests a machine credential, but if that fails, it asks for the OPEN user's credential for SETCLIENTID.  The retried upcall request is for "service=<null> UID=nnnn".
> 
> If root kinit's for the mount, but the user does not kinit before accessing the mount, the SETCLIENTID still fails!  Root's ticket and "gssd -n" has nothing to do with SETCLIENTID's security flavor, as far as I can tell.
> 
> With 3.7, the SETCLIENTID is done right at mount time.  There is no user with credentials trying to do an OPEN, so the only credential the kernel tries is the machine's credential.  Perhaps the kernel could retry the upcall with "server=<null> UID=0".
> 
> >> It is possible that with 3.10, he might get away with doing lease management
> >> with AUTH_SYS, but that seems unlikely.  Given that an integrity protecting
> >> security flavour is now recommended for SETCLIENTID, his server admin
> >> could easily make a case that AUTH_SYS won't be accepted.  In that case he
> >> would still have a problem.
> > 
> > His problem then would be with his server admin, not with us.
> > 
> > If a server admin wants to protect his server by denying the use of AUTH_SYS for lease management, he has every right to do that.  But he must also therefore understand the position that puts his users in.
> > 
> >> Given that "gssd -n" has meant "no machine credential needed" in the past,
> > 
> > It never meant that.  It meant "use root's credential as the machine credential".  I think we should be clear about exactly what "-n" is.
> 
> Right, I should heed my own words.
> 
> > I also believe "-n" (or "-N") is a workaround.  Logging in as root and authenticating is a pain for users, who have already logged in once.
> 
> This is still true, however.
> 
> >> I think it is important to preserve the possibility of running secure NFS with
> >> no machine credential.  I don't see how your changes (valuable though they are)
> >> achieve that.
> > 
> > Please explain how allowing lease management to occur using AUTH_SYS is somehow a major new exposure.  A mixture of sec=sys and sec=krb5 mounts today means there's a good chance that a sec=sys mount might occur first, and thus lease management uses AUTH_SYS anyway.
> > 
> > Can you share your attack model?  Why do you believe that using AUTH_SYS for lease management on sec=krb5 mounts will be unacceptable to users?  I suspect that most will not notice or care.
> 
> I'm still interested in knowing if we have a palpable hole in the current solution in 3.10.
> 
> >> So I think we either need to change "gssd -n" to mean "never use machine
> >> credentials" or add something like "gssd -N", so the two different possible
> >> uses of machine credentials can be separately controlled.
> > 
> > I'm not convinced that "-N" is the right way to fix this.  Another knob to turn means more testing for us, more documentation to write and maintain, and so on.  It doesn't get us any farther forward.  The difference between "-n" (which no longer works) and "-N" is subtle and confusing.  Explaining when to use "-N" versus "-n" is not a job I would like to do, and I'm not interested in taking the support calls either.  :-p
> 
> Let me restate this without all the cleverness.
> 
> Trond's recommendation was to tell gssd exactly where to find the machine credential if it's not in a keytab: basically an option that says "look in this file for that user credential".  That was our first try, but it was clumsy.
> 
> You want "gssd -N" to mean specifically "use root's ticket as the machine credential."  Then "-n -N" would mean "force root to kinit, and use root's ticket as the machine credential."  (Perhaps "-N" doesn't make sense unless "-n" is also specified...?)
> 
> But that's still not the same as 3.6's behavior.  And, it would still be a regression, as I understand it, because a user space change would be needed after a kernel upgrade.  For that reason, for 3.[789] we probably want to find a kernel-only solution.
> 
> Command line options on gssd are really poor from a usability standpoint.  IMO in the long run we want "mount sec=krb5 without a client keytab" just to work.  No monkeying with gssd.
> 

Hi Chuck,
 thanks for you explanations and investigations - they certainly help.

Let me try to present my perspective.

The "problem" situation is where the server admin does not want to hand out
machine credentials (keytabs).
As you say:
   "Handing out keytabs for every NFS client that wants to use Kerberized NFS
    is onerous for some administrators"

If I want to use my personal machine to access only my files over NFS then,
from the server admin's perspective, a kerberos login should be enough.  And
until recently it has been.

This server admin might have read Dave Noveck's recommendations that you
quote in commit 4edaa308888b:

    Dave Noveck's migration issues draft recommends the use of an
    integrity-protecting security flavor for the SETCLIENTID operation.

and decided to disable AUTH_SYS for SETCLIENTID.

For 3.6 (and earlier?) a client could work with this by running "kinit" as
both themselves and root, and running "gssd -n".  Then whenever they access
files, or whenever root accesses files on their behalf, the access uses their
credential and works.  The only access that tried to use the machine
credential (which doesn't exist) would be performed in the context of an
access by the user (an open) and would fall back on the user's credential.

In 3.[789] the mount doesn't work because it requires a SETCLIENTID which
tries to use the machine credential and has no user to fall back on.
It falls back on AUTH_NONE (is that right?) which the server doesn't accept.

In 3.10 the same happens but now it falls back on AUTH_SYS and the server
still doesn't accept it (because Dave said integrity protection is important).

The only way I can see out of this is for the SETCLIENTID request at mount
time to be able to use root's credential.

I don't much care whether:
  a- the kernel requests
          mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2
     if
          mech=krb5 uid=0 service=* enctypes=18,17,16,23,3,1,2
     fails, or
  b- gssd tried the credential for 'uid=0' when a lookup for the machine
     credential fails, or
  c- "gssd -N" means "use uid=0 credential when kernel requests a machine
     credential" or
  d- "gssd -n" adds the above to its current meaning.

'c' is what I implemented as it has least impact on people who don't care.
I think you suggested 'a' above.  I'm certainly happy with that, but I'd
rather not try to implement it myself.
I don't particularly need a "kernel-only solution" as I can update nfs-utils
in openSUSE just as easily as the kernel, but it is a good idea in principle
and I'd be happy to get it tested if someone else implements it :-)


Your desire to make it "just work" without monkeying around with flags to gssd
is certainly good.  However there is a key difference between a multi-user
client and a single-user client.  For the multi-user client there must be a
separate "root" identity, whether in the form of a machine credential or a
credential for uid=0.  For the single-user client there is only one identity.
So somewhere that distinction needs to be configured.
One way is requiring a 'kinit' as root and running "gssd -n", but that is
indeed clumsy.
The only other approach I can think if is a mount option: 'root-uid=NN'
This would mean that any access that would require a "uid=0" credential
lookup instead uses a "uid=NN" credential lookup.  That would mean that the
single user would not need to "kinit" as root.  They just "kinit" as
themselves and mount the filesystem with the right option and everything
"just works".

Thanks,
NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [PATCH 0/3] Various gssd fixes including machine-credential issue.

[Chuck Lever]

On Jun 4, 2013, at 9:26 PM, NeilBrown <neilb@suse.de> wrote:

> On Tue, 4 Jun 2013 15:16:01 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:
> 
>> 
>> On Jun 3, 2013, at 9:13 PM, Chuck Lever <chuck.lever@oracle.com> wrote:
>> 
>>> 
>>> On Jun 3, 2013, at 7:30 PM, NeilBrown <neilb@suse.de> wrote:
>>> 
>>>> On Mon, 3 Jun 2013 00:32:54 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:
>>>> 
>>>>> 
>>>>> On Jun 2, 2013, at 11:01 PM, NeilBrown <neilb@suse.de> wrote:
>>>>> 
>>>>>> On Sun, 2 Jun 2013 22:45:16 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:
>>>>>> 
>>>>>>> 
>>>>>>> On Jun 2, 2013, at 10:23 PM, NeilBrown <neilb@suse.de> wrote:
>>>>>>> 
>>>>>>>> On Sun, 2 Jun 2013 22:01:50 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Jun 2, 2013, at 9:00 PM, Neil Brown <neilb@suse.de> wrote:
>>>>>>>>> 
>>>>>>>>>> As you probably know, since 3.7 (I think) Linux NFS has explicitly
>>>>>>>>>> asked for machine credentials for certain requests rather than asking
>>>>>>>>>> for root credentials as is previously did.
>>>>>>>>>> This causes a regression for people who don't have any machine
>>>>>>>>>> credentials configured and use "gssd -n".
>>>>>>>>>> 
>>>>>>>>>> I gather this was discussed on the mailing list earlier this year but
>>>>>>>>>> not resolved.
>>>>>>>>> 
>>>>>>>>> It's resolved in 3.10-rc.
>>>>>>>>> 
>>>>>>>>> The kernel will attempt to use krb5i for lease management operations.  If that fails because there is no keytab available, it falls back to using AUTH_SYS.
>>>>>>>> 
>>>>>>>> And if the server refuses to accept AUTH_SYS?
>>>>>>>> 
>>>>>>>> I guess this is commit 79d852bf5e7691dc7 ??
>>>>>>> 
>>>>>>> That's one of the subsequent bug fixes.  The initial change is commit 4edaa308.
>>>>>>> 
>>>>>>>> It seems to say that the server should always accept AUTH_SYS ... is that right?
>>>>>>> 
>>>>>>> If we ever find a server implementation that does not support either Kerberos or AUTH_SYS, we can add another step to the negotiation.
>>>>>>> 
>>>>>>> So far, despite RFC 3530 not requiring AUTH_SYS support on NFSv4 servers, I haven't found an implementation that does not support AUTH_SYS.  We have found one (FreeBSD) that does not support AUTH_NONE.  We do know that some servers allow administrators to control what security flavors are allowed for lease management.
>>>>>>> 
>>>>>>>> That commit isn't tagged for -stable.
>>>>>>>> So do we still need to make it work for 3.7,3.8,3.9 users?
>>>>>>> 
>>>>>>> There are several commits that would need to be back-ported, starting with commit 4edaa308.  I am not certain they would apply cleanly to 3.[789], but a backport should not be difficult.
>>>>>>> 
>>>>>>> This change also requires that now gssd must be running on the client.  Otherwise without gssd a sec=sys mount hangs for a bit waiting for the upcall to time out (since the client will attempt to use krb5i for lease management operations).  Trond and Bruce have been discussing a change to address that.
>>>>>> 
>>>>>> Thanks for the explanation.  That all looks rather painful to back-port
>>>>>> though, especially as some of it isn't even written yet :-)
>>>>>> I think I'll stick with my "-N" option for openSUSE for now.
>>>>>> 
>>>>>> Do you think that supporting -N (or similar) so that the admin can ask for
>>>>>> root credentials to be used for SETCLIENTID requests is reasonable? i.e. what
>>>>>> do you think of my patch going in to nfs-utils anyway?
>>>>> 
>>>>> So, let me confirm my understanding of your proposed changes: a user logs in as root, then kinit's with her own user principal.  That establishes a Kerberos credential for root to use, and "-N" makes gssd return this credential when the kernel requests "service=*".
>>>> 
>>>> Correct.
>>>> 
>>>>> 
>>>>> Lease management is shared among all NFS users on a client.  In particular, machine credentials give us two important features that cannot be matched by user credentials:
>>>>> 
>>>>> o  Machine credentials never expire even when users log out.  On a
>>>>>   multi-user client, you don't want credential expiry or a user
>>>>>   logout to cause lease management to stop working for other users. 
>>>> 
>>>> ... while on a single-user client, this isn't really an issue.
>>>> 
>>>> And I can easily imagine a server admin not wanting to hand out credentials
>>>> that never expire, but still requiring strong credentials for lease
>>>> management.
>>> 
>>> Why, exactly, would strong credentials be required for lease management, but an admin would choose not to hand out keytabs?
>>> 
>>>> 
>>>>> 
>>>>> o  Machine credentials are always the same no matter who is logged
>>>>>   in.  The client has to use the same credential every time for
>>>>>   lease management, or else servers return CLID_INUSE and prevent a
>>>>>   fresh lease from being established (at least until the existing
>>>>>   lease for that clientid expires on the server).
>>>> 
>>>> Again, not really an issue for a single-user client.  And single-user
>>>> machines are certainly a common use-case these days.
>>>> 
>>>>> 
>>>>> Using root's credentials for lease management makes it likely one of these two bullets will end up in your foot.  Specifically for the use case where one user is always using the same client (say, a student's laptop), "-N" might work fine.  For other use cases, like a shared build machine that doesn't have a keytab, "-N" would not work reliably.
>>>> 
>>>> Completely agree.  So -N should certainly not be the default.
>>>> 
>>>>> 
>>>>> With 3.10, if we can't use a Kerberos host principal for lease management, we'll use a less secure equivalent.  The point of backing off the security flavor for lease management is that AUTH_SYS with UID 0, or even AUTH_NONE, are essentially machine credentials.  They don't expire, and the client will present a consistent credential for lease management, no matter what user logs in or leaves the machine.
>>>>> 
>>>>> So, yes, machine credentials are a pain in the ass, and have been for years.  I'm sorry this took 3 kernel releases to figure out, but I'm hoping that the path we're on with these kernel changes is towards more transparent support for a wider array of use cases.  NFSv4.1 also offers some relief in this area (see SP4_MACH_CRED, which I think we do not support quite yet).
>>>> 
>>>> All sounds very good, but doesn't seem to address the issue.
>>> 
>>>> For the openSUSE user who raised this, he simply does not have a machine
>>>> credential. I don't know why, maybe the server admin doesn't want to hand
>>>> them out.
>>> 
>>> Handing out keytabs for every NFS client that wants to use Kerberized NFS is onerous for some administrators.
>>> 
>>>> But this hasn't been a problem because "gssd -n" meant that a
>>>> machine credential would never be used.
>>> 
>>> No, it meant that root's user credential was used instead of a service credential for lease  management.  As I explained above, that solution has some compromises for some important use cases.
>> 
>> Correction:
>> 
>> rpc.gssd(8) on Fedora 18 says:
>> 
>>> With the -n option, "machine credentials" will not be used for accesses by UID 0.
>> 
>> 
>> In other words, normally root doesn't have a Kerberos ticket.  The kernel uses the machine's credentials for UID 0.  "-n" means only that root has to kinit.
>> 
>> 
>>>> But now,  with 3.[789], "gssd -n" still wants the machine credential that
>>>> doesn't exist.
>>> 
>>> The change that introduced the requirement for a machine credential is commit 68c97153 "SUNRPC: Clean up the RPCSEC_GSS service ticket requests", Tue Jan 3 13:22:46 2012, which was merged long before 3.7.
>>> 
>>> The Uniform Client String changes in 3.7 merely made the machine credential requirement more obtrusive.  The new broken "-n" behavior is an unintended side effect.
>> 
>> I was wondering what exactly about 3.7's UCS changes exposed the requirement for a machine credential, so I built a 3.6 kernel and tried it.
>> 
>> The key difference between 3.6 and 3.7 is that 3.6 performs the SETCLIENTID much later, after a user has logged in and kinit'd, and is actually doing an OPEN.
>> 
>> 3.6 requests a machine credential, but if that fails, it asks for the OPEN user's credential for SETCLIENTID.  The retried upcall request is for "service=<null> UID=nnnn".
>> 
>> If root kinit's for the mount, but the user does not kinit before accessing the mount, the SETCLIENTID still fails!  Root's ticket and "gssd -n" has nothing to do with SETCLIENTID's security flavor, as far as I can tell.
>> 
>> With 3.7, the SETCLIENTID is done right at mount time.  There is no user with credentials trying to do an OPEN, so the only credential the kernel tries is the machine's credential.  Perhaps the kernel could retry the upcall with "server=<null> UID=0".
>> 
>>>> It is possible that with 3.10, he might get away with doing lease management
>>>> with AUTH_SYS, but that seems unlikely.  Given that an integrity protecting
>>>> security flavour is now recommended for SETCLIENTID, his server admin
>>>> could easily make a case that AUTH_SYS won't be accepted.  In that case he
>>>> would still have a problem.
>>> 
>>> His problem then would be with his server admin, not with us.
>>> 
>>> If a server admin wants to protect his server by denying the use of AUTH_SYS for lease management, he has every right to do that.  But he must also therefore understand the position that puts his users in.
>>> 
>>>> Given that "gssd -n" has meant "no machine credential needed" in the past,
>>> 
>>> It never meant that.  It meant "use root's credential as the machine credential".  I think we should be clear about exactly what "-n" is.
>> 
>> Right, I should heed my own words.
>> 
>>> I also believe "-n" (or "-N") is a workaround.  Logging in as root and authenticating is a pain for users, who have already logged in once.
>> 
>> This is still true, however.
>> 
>>>> I think it is important to preserve the possibility of running secure NFS with
>>>> no machine credential.  I don't see how your changes (valuable though they are)
>>>> achieve that.
>>> 
>>> Please explain how allowing lease management to occur using AUTH_SYS is somehow a major new exposure.  A mixture of sec=sys and sec=krb5 mounts today means there's a good chance that a sec=sys mount might occur first, and thus lease management uses AUTH_SYS anyway.
>>> 
>>> Can you share your attack model?  Why do you believe that using AUTH_SYS for lease management on sec=krb5 mounts will be unacceptable to users?  I suspect that most will not notice or care.
>> 
>> I'm still interested in knowing if we have a palpable hole in the current solution in 3.10.
>> 
>>>> So I think we either need to change "gssd -n" to mean "never use machine
>>>> credentials" or add something like "gssd -N", so the two different possible
>>>> uses of machine credentials can be separately controlled.
>>> 
>>> I'm not convinced that "-N" is the right way to fix this.  Another knob to turn means more testing for us, more documentation to write and maintain, and so on.  It doesn't get us any farther forward.  The difference between "-n" (which no longer works) and "-N" is subtle and confusing.  Explaining when to use "-N" versus "-n" is not a job I would like to do, and I'm not interested in taking the support calls either.  :-p
>> 
>> Let me restate this without all the cleverness.
>> 
>> Trond's recommendation was to tell gssd exactly where to find the machine credential if it's not in a keytab: basically an option that says "look in this file for that user credential".  That was our first try, but it was clumsy.
>> 
>> You want "gssd -N" to mean specifically "use root's ticket as the machine credential."  Then "-n -N" would mean "force root to kinit, and use root's ticket as the machine credential."  (Perhaps "-N" doesn't make sense unless "-n" is also specified...?)
>> 
>> But that's still not the same as 3.6's behavior.  And, it would still be a regression, as I understand it, because a user space change would be needed after a kernel upgrade.  For that reason, for 3.[789] we probably want to find a kernel-only solution.
>> 
>> Command line options on gssd are really poor from a usability standpoint.  IMO in the long run we want "mount sec=krb5 without a client keytab" just to work.  No monkeying with gssd.
>> 
> 
> Hi Chuck,
> thanks for you explanations and investigations - they certainly help.
> 
> Let me try to present my perspective.
> 
> The "problem" situation is where the server admin does not want to hand out
> machine credentials (keytabs).
> As you say:
>   "Handing out keytabs for every NFS client that wants to use Kerberized NFS
>    is onerous for some administrators"

We agree that there are environments where no host or nfs service principal is available for clients, and we agree that type of environment should be supported with Linux's NFSv4 client.

Where we differ is what compromises we will accept in that type of environment.

> If I want to use my personal machine to access only my files over NFS then,
> from the server admin's perspective, a kerberos login should be enough.  And
> until recently it has been.

I agree that not supporting this is a bug (regression) and should be fixed.

> This server admin might have read Dave Noveck's recommendations that you
> quote in commit 4edaa308888b:
> 
>    Dave Noveck's migration issues draft recommends the use of an
>    integrity-protecting security flavor for the SETCLIENTID operation.

That's a recommendation, not a requirement.  This means the standards authors (me, being one of the authors on Dave's migration draft) recognize there are legitimate use cases where using an integrity-protecting security flavor is not feasible.

Either the client or the server can enforce the use of an integrity-protecting flavor, but it is also possible that neither do.  (AUTH_NONE and AUTH_SYS-only environments being one important example).

> and decided to disable AUTH_SYS for SETCLIENTID.

Again, I ask: what is being protected?  Why would an administrator make this requirement?  I would like to clearly understand this particular attack so we can design a smart solution that does what administrators want.

> For 3.6 (and earlier?) a client could work with this by running "kinit" as
> both themselves and root, and running "gssd -n".  Then whenever they access
> files, or whenever root accesses files on their behalf, the access uses their
> credential and works.  The only access that tried to use the machine
> credential (which doesn't exist) would be performed in the context of an
> access by the user (an open) and would fall back on the user's credential.
> 
> In 3.[789] the mount doesn't work because it requires a SETCLIENTID which
> tries to use the machine credential and has no user to fall back on.
> It falls back on AUTH_NONE (is that right?) which the server doesn't accept.

The 3.7+ fallback logic works just like 3.6 and earlier.  The difference is that now the SETCLIENTID occurs as the first operation on the first mount.  At that early stage, there are no user credentials available because no-one has attempted to create any open or lock state, which is what is mined on the client for a user credential.

> In 3.10 the same happens but now it falls back on AUTH_SYS and the server
> still doesn't accept it (because Dave said integrity protection is important).

No one has proposed a server change, so AUTH_SYS should be a reasonable choice.  Servers do not have to enforce the use of integrity-protecting security flavors for lease management unless the administrators recognize a palpable threat.  I'm trying to identify what kind of threats that might be, because so far we have been talking in hypotheticals.

> The only way I can see out of this is for the SETCLIENTID request at mount
> time to be able to use root's credential.

That's what the patch I posted does.  If the kernel requests "service=*" and gssd returns -EACCES, the kernel will look for a user credential.  If there are no user credentials to try, it will try "service=<null> UID=0".  If that doesn't work, 3.[789] fail.

I haven't worked out how 3.10+ will behave.  I suspect it could try using the UID=0 principal to establish a client ID using krb5i, and if _that_ fails, then slip back to AUTH_SYS.

> I don't much care whether:
>  a- the kernel requests
>          mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2
>     if
>          mech=krb5 uid=0 service=* enctypes=18,17,16,23,3,1,2
>     fails, or
>  b- gssd tried the credential for 'uid=0' when a lookup for the machine
>     credential fails, or
>  c- "gssd -N" means "use uid=0 credential when kernel requests a machine
>     credential" or
>  d- "gssd -n" adds the above to its current meaning.
> 
> 'c' is what I implemented as it has least impact on people who don't care.
> I think you suggested 'a' above.  I'm certainly happy with that, but I'd
> rather not try to implement it myself.
> I don't particularly need a "kernel-only solution" as I can update nfs-utils
> in openSUSE just as easily as the kernel,

Understood, distributors are lucky in that regard ;-) .  The requirement for a kernel-only solution is for upstream.  Upgrading from 3.6 to 3.7 should not require a user space adjustment like a new command line option or a new version of gssd.

> but it is a good idea in principle
> and I'd be happy to get it tested if someone else implements it :-)

> Your desire to make it "just work" without monkeying around with flags to gssd
> is certainly good.  However there is a key difference between a multi-user
> client and a single-user client.  For the multi-user client there must be a
> separate "root" identity, whether in the form of a machine credential or a
> credential for uid=0.  For the single-user client there is only one identity.
> So somewhere that distinction needs to be configured.

I believe we will be fundamentally better off if we use the same mechanism in both cases.  The implementation is less complex and easier for us to test and maintain.  The promise of a broad "no configuration" solution also has its appeal.

> One way is requiring a 'kinit' as root and running "gssd -n", but that is
> indeed clumsy.
> The only other approach I can think if is a mount option: 'root-uid=NN'
> This would mean that any access that would require a "uid=0" credential
> lookup instead uses a "uid=NN" credential lookup.  That would mean that the
> single user would not need to "kinit" as root.  They just "kinit" as
> themselves and mount the filesystem with the right option and everything
> "just works".

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: [PATCH 0/3] Various gssd fixes including machine-credential issue.

[Chuck Lever]

On Jun 5, 2013, at 11:37 AM, Chuck Lever <chuck.lever@oracle.com> wrote:

> 
> On Jun 4, 2013, at 9:26 PM, NeilBrown <neilb@suse.de> wrote:
>> 
>> Hi Chuck,
>> thanks for you explanations and investigations - they certainly help.
>> 
>> Let me try to present my perspective.
>> 
>> The "problem" situation is where the server admin does not want to hand out
>> machine credentials (keytabs).
>> As you say:
>>  "Handing out keytabs for every NFS client that wants to use Kerberized NFS
>>   is onerous for some administrators"
> 
> We agree that there are environments where no host or nfs service principal is available for clients, and we agree that type of environment should be supported with Linux's NFSv4 client.
> 
> Where we differ is what compromises we will accept in that type of environment.
> 
>> If I want to use my personal machine to access only my files over NFS then,
>> from the server admin's perspective, a kerberos login should be enough.  And
>> until recently it has been.
> 
> I agree that not supporting this is a bug (regression) and should be fixed.
> 
>> This server admin might have read Dave Noveck's recommendations that you
>> quote in commit 4edaa308888b:
>> 
>>   Dave Noveck's migration issues draft recommends the use of an
>>   integrity-protecting security flavor for the SETCLIENTID operation.
> 
> That's a recommendation, not a requirement.  This means the standards authors (me, being one of the authors on Dave's migration draft) recognize there are legitimate use cases where using an integrity-protecting security flavor is not feasible.
> 
> Either the client or the server can enforce the use of an integrity-protecting flavor, but it is also possible that neither do.  (AUTH_NONE and AUTH_SYS-only environments being one important example).
> 
>> and decided to disable AUTH_SYS for SETCLIENTID.
> 
> Again, I ask: what is being protected?  Why would an administrator make this requirement?  I would like to clearly understand this particular attack so we can design a smart solution that does what administrators want.
> 
>> For 3.6 (and earlier?) a client could work with this by running "kinit" as
>> both themselves and root, and running "gssd -n".  Then whenever they access
>> files, or whenever root accesses files on their behalf, the access uses their
>> credential and works.  The only access that tried to use the machine
>> credential (which doesn't exist) would be performed in the context of an
>> access by the user (an open) and would fall back on the user's credential.
>> 
>> In 3.[789] the mount doesn't work because it requires a SETCLIENTID which
>> tries to use the machine credential and has no user to fall back on.
>> It falls back on AUTH_NONE (is that right?) which the server doesn't accept.
> 
> The 3.7+ fallback logic works just like 3.6 and earlier.  The difference is that now the SETCLIENTID occurs as the first operation on the first mount.  At that early stage, there are no user credentials available because no-one has attempted to create any open or lock state, which is what is mined on the client for a user credential.
> 
>> In 3.10 the same happens but now it falls back on AUTH_SYS and the server
>> still doesn't accept it (because Dave said integrity protection is important).
> 
> No one has proposed a server change, so AUTH_SYS should be a reasonable choice.  Servers do not have to enforce the use of integrity-protecting security flavors for lease management unless the administrators recognize a palpable threat.  I'm trying to identify what kind of threats that might be, because so far we have been talking in hypotheticals.
> 
>> The only way I can see out of this is for the SETCLIENTID request at mount
>> time to be able to use root's credential.
> 
> That's what the patch I posted does.  If the kernel requests "service=*" and gssd returns -EACCES, the kernel will look for a user credential.  If there are no user credentials to try, it will try "service=<null> UID=0".  If that doesn't work, 3.[789] fail.
> 
> I haven't worked out how 3.10+ will behave.  I suspect it could try using the UID=0 principal to establish a client ID using krb5i, and if _that_ fails, then slip back to AUTH_SYS.
> 
>> I don't much care whether:
>> a- the kernel requests
>>         mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2
>>    if
>>         mech=krb5 uid=0 service=* enctypes=18,17,16,23,3,1,2
>>    fails, or
>> b- gssd tried the credential for 'uid=0' when a lookup for the machine
>>    credential fails, or
>> c- "gssd -N" means "use uid=0 credential when kernel requests a machine
>>    credential" or
>> d- "gssd -n" adds the above to its current meaning.
>> 
>> 'c' is what I implemented as it has least impact on people who don't care.
>> I think you suggested 'a' above.  I'm certainly happy with that, but I'd
>> rather not try to implement it myself.
>> I don't particularly need a "kernel-only solution" as I can update nfs-utils
>> in openSUSE just as easily as the kernel,
> 
> Understood, distributors are lucky in that regard ;-) .  The requirement for a kernel-only solution is for upstream.  Upgrading from 3.6 to 3.7 should not require a user space adjustment like a new command line option or a new version of gssd.

In terms of providing fixes for this issue:

  o  3.7.y and 3.8.y are now EOL.  I think I should at least post patches here for these kernels so a fix is available for distributions that continue to rely on these kernel versions

  o  3.9.y is still maintained.  The 3.7 fix I posted yesterday will need to be ported, but I'm guessing that it will work for 3.9 without a lot of alteration

  o  A fix for 3.10-rc will need some adjustment, since there is a significant change in that kernel to use krb5i when possible for lease management operations.  Or, should we consider not applying this fix to upstream?  Would Greg take a fix for 3.9.y that hasn't gone into the upstream kernels?

Thoughts?

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: [PATCH 0/3] Various gssd fixes including machine-credential issue.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 7251 bytes --]

On Wed, 5 Jun 2013 13:14:00 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:

> 
> On Jun 5, 2013, at 11:37 AM, Chuck Lever <chuck.lever@oracle.com> wrote:
> 
> > 
> > On Jun 4, 2013, at 9:26 PM, NeilBrown <neilb@suse.de> wrote:
> >> 
> >> Hi Chuck,
> >> thanks for you explanations and investigations - they certainly help.
> >> 
> >> Let me try to present my perspective.
> >> 
> >> The "problem" situation is where the server admin does not want to hand out
> >> machine credentials (keytabs).
> >> As you say:
> >>  "Handing out keytabs for every NFS client that wants to use Kerberized NFS
> >>   is onerous for some administrators"
> > 
> > We agree that there are environments where no host or nfs service principal is available for clients, and we agree that type of environment should be supported with Linux's NFSv4 client.
> > 
> > Where we differ is what compromises we will accept in that type of environment.
> > 
> >> If I want to use my personal machine to access only my files over NFS then,
> >> from the server admin's perspective, a kerberos login should be enough.  And
> >> until recently it has been.
> > 
> > I agree that not supporting this is a bug (regression) and should be fixed.
> > 
> >> This server admin might have read Dave Noveck's recommendations that you
> >> quote in commit 4edaa308888b:
> >> 
> >>   Dave Noveck's migration issues draft recommends the use of an
> >>   integrity-protecting security flavor for the SETCLIENTID operation.
> > 
> > That's a recommendation, not a requirement.  This means the standards authors (me, being one of the authors on Dave's migration draft) recognize there are legitimate use cases where using an integrity-protecting security flavor is not feasible.
> > 
> > Either the client or the server can enforce the use of an integrity-protecting flavor, but it is also possible that neither do.  (AUTH_NONE and AUTH_SYS-only environments being one important example).
> > 
> >> and decided to disable AUTH_SYS for SETCLIENTID.
> > 
> > Again, I ask: what is being protected?  Why would an administrator make this requirement?  I would like to clearly understand this particular attack so we can design a smart solution that does what administrators want.
> > 
> >> For 3.6 (and earlier?) a client could work with this by running "kinit" as
> >> both themselves and root, and running "gssd -n".  Then whenever they access
> >> files, or whenever root accesses files on their behalf, the access uses their
> >> credential and works.  The only access that tried to use the machine
> >> credential (which doesn't exist) would be performed in the context of an
> >> access by the user (an open) and would fall back on the user's credential.
> >> 
> >> In 3.[789] the mount doesn't work because it requires a SETCLIENTID which
> >> tries to use the machine credential and has no user to fall back on.
> >> It falls back on AUTH_NONE (is that right?) which the server doesn't accept.
> > 
> > The 3.7+ fallback logic works just like 3.6 and earlier.  The difference is that now the SETCLIENTID occurs as the first operation on the first mount.  At that early stage, there are no user credentials available because no-one has attempted to create any open or lock state, which is what is mined on the client for a user credential.
> > 
> >> In 3.10 the same happens but now it falls back on AUTH_SYS and the server
> >> still doesn't accept it (because Dave said integrity protection is important).
> > 
> > No one has proposed a server change, so AUTH_SYS should be a reasonable choice.  Servers do not have to enforce the use of integrity-protecting security flavors for lease management unless the administrators recognize a palpable threat.  I'm trying to identify what kind of threats that might be, because so far we have been talking in hypotheticals.
> > 
> >> The only way I can see out of this is for the SETCLIENTID request at mount
> >> time to be able to use root's credential.
> > 
> > That's what the patch I posted does.  If the kernel requests "service=*" and gssd returns -EACCES, the kernel will look for a user credential.  If there are no user credentials to try, it will try "service=<null> UID=0".  If that doesn't work, 3.[789] fail.
> > 
> > I haven't worked out how 3.10+ will behave.  I suspect it could try using the UID=0 principal to establish a client ID using krb5i, and if _that_ fails, then slip back to AUTH_SYS.
> > 
> >> I don't much care whether:
> >> a- the kernel requests
> >>         mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2
> >>    if
> >>         mech=krb5 uid=0 service=* enctypes=18,17,16,23,3,1,2
> >>    fails, or
> >> b- gssd tried the credential for 'uid=0' when a lookup for the machine
> >>    credential fails, or
> >> c- "gssd -N" means "use uid=0 credential when kernel requests a machine
> >>    credential" or
> >> d- "gssd -n" adds the above to its current meaning.
> >> 
> >> 'c' is what I implemented as it has least impact on people who don't care.
> >> I think you suggested 'a' above.  I'm certainly happy with that, but I'd
> >> rather not try to implement it myself.
> >> I don't particularly need a "kernel-only solution" as I can update nfs-utils
> >> in openSUSE just as easily as the kernel,
> > 
> > Understood, distributors are lucky in that regard ;-) .  The requirement for a kernel-only solution is for upstream.  Upgrading from 3.6 to 3.7 should not require a user space adjustment like a new command line option or a new version of gssd.
> 
> In terms of providing fixes for this issue:
> 
>   o  3.7.y and 3.8.y are now EOL.  I think I should at least post patches here for these kernels so a fix is available for distributions that continue to rely on these kernel versions
> 
>   o  3.9.y is still maintained.  The 3.7 fix I posted yesterday will need to be ported, but I'm guessing that it will work for 3.9 without a lot of alteration
> 
>   o  A fix for 3.10-rc will need some adjustment, since there is a significant change in that kernel to use krb5i when possible for lease management operations.  Or, should we consider not applying this fix to upstream?  Would Greg take a fix for 3.9.y that hasn't gone into the upstream kernels?
> 
> Thoughts?

I don't think Greg is keen on patches that aren't mainline, but he is a
reasonable man and will listen to a rational argument.
However I think you would need a clear idea of what *is* going to be in 3.10
before offering an alternate version of it for 3.9.
If all known servers really do accept SETCLIENTID with AUTH_SYS in all cases
then I guess you don't need anything new for mainline.
If it is then easier to make 3.9 fall back to
     "mech=krb5 uid=0 service=<null>"
than to have it fall back to AUTH_SYS, then it seems justifiable that the
"backport" to 3.9 does just that.

i.e argument to Greg is:
     3.9 fails if a machine credential is not available.  3.10 will now fall
     back to using AUTH_SYS.  In 3.9 it is easier to fall back on using the
     uid=0 credential and as we have always required that either the machine
     credential or the uid=0 credential exist (for gss mounts), do that is
     safe.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [PATCH 0/3] Various gssd fixes including machine-credential issue.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3747 bytes --]

On Wed, 5 Jun 2013 11:37:12 -0400 Chuck Lever <chuck.lever@oracle.com> wrote:

> Again, I ask: what is being protected?  Why would an administrator make this requirement?  I would like to clearly understand this particular attack so we can design a smart solution that does what administrators want.

I honestly have no idea.  But if there is nothing to be protected, why would
someone recommend (even if they don't require) integrity protection?

If I was setting up a server and was concerned about security I would be
choosing options that required signed messages where-ever possible, even if I
couldn't see the exact scenario by which a non-signed message could cause a
problem.

Certainly in an AUTH_SYS environment (trusted network transport), AUTH_SYS or
AUTH_NONE should be enough for SETCLIENTID.  But you want an AUTH_GSS
environment, why not aim to make everything use AUTH_GSS?

> 
> > For 3.6 (and earlier?) a client could work with this by running "kinit" as
> > both themselves and root, and running "gssd -n".  Then whenever they access
> > files, or whenever root accesses files on their behalf, the access uses their
> > credential and works.  The only access that tried to use the machine
> > credential (which doesn't exist) would be performed in the context of an
> > access by the user (an open) and would fall back on the user's credential.
> > 
> > In 3.[789] the mount doesn't work because it requires a SETCLIENTID which
> > tries to use the machine credential and has no user to fall back on.
> > It falls back on AUTH_NONE (is that right?) which the server doesn't accept.
> 
> The 3.7+ fallback logic works just like 3.6 and earlier.  The difference is that now the SETCLIENTID occurs as the first operation on the first mount.  At that early stage, there are no user credentials available because no-one has attempted to create any open or lock state, which is what is mined on the client for a user credential.
> 
> > In 3.10 the same happens but now it falls back on AUTH_SYS and the server
> > still doesn't accept it (because Dave said integrity protection is important).
> 
> No one has proposed a server change, so AUTH_SYS should be a reasonable choice.  Servers do not have to enforce the use of integrity-protecting security flavors for lease management unless the administrators recognize a palpable threat.  I'm trying to identify what kind of threats that might be, because so far we have been talking in hypotheticals.

If it is only hypothetical, why is it even recommended?  I must be missing
something.
If we expect the server will always accept AUTH_SYS, why ever bother sending
anything else?
If we expect the server might not accept AUTH_SYS, we should do our best to
send the appropriate authentication.



> > Your desire to make it "just work" without monkeying around with flags to gssd
> > is certainly good.  However there is a key difference between a multi-user
> > client and a single-user client.  For the multi-user client there must be a
> > separate "root" identity, whether in the form of a machine credential or a
> > credential for uid=0.  For the single-user client there is only one identity.
> > So somewhere that distinction needs to be configured.
> 
> I believe we will be fundamentally better off if we use the same mechanism in both cases.  The implementation is less complex and easier for us to test and maintain.  The promise of a broad "no configuration" solution also has its appeal.

Certainly agree it would be better.  Not convinced it is actually possible.
Not an important issue for me at the moment though.

I've provided a patched kernel to customer.  Hopefully will hear back soon.

Thanks
NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [PATCH 0/3] Various gssd fixes including machine-credential issue.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 755 bytes --]

On Thu, 6 Jun 2013 09:43:36 +1000 NeilBrown <neilb@suse.de> wrote:

> I've provided a patched kernel to customer.  Hopefully will hear back soon.

Hi Chuck,
 the patch appears to work exactly as advertised and completely resolves the
 issue - thanks.

Getting back to the idea of making this sort of thing work seamlessly,
another possibility to consider is to use the *real* uid  of the process
which performs the mount do authenticate SETCLIENTID - either as the primary
authentication or as a fall-back.
Then the a mountpoint with "users" listed as an option in /etc/fstab can be
mounted by anyone with appropriate credentials and no credential should be
needed for root, and not machine credential should be needed either.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [PATCH 0/3] Various gssd fixes including machine-credential issue.

[Chuck Lever]

On Jun 12, 2013, at 2:12 AM, NeilBrown <neilb@suse.de> wrote:

> On Thu, 6 Jun 2013 09:43:36 +1000 NeilBrown <neilb@suse.de> wrote:
> 
>> I've provided a patched kernel to customer.  Hopefully will hear back soon.
> 
> Hi Chuck,
> the patch appears to work exactly as advertised and completely resolves the
> issue - thanks.

Thanks to you and your customer for confirming the approach.

Unfortunately Trond NAK'd a for-3.10 patch that uses RPC_TASK_ROOTCREDS.  I posted a patch Friday that takes a different approach to acquiring the root user's credentials that should behave like the old patch in every way, and be easier to back-port.

> Getting back to the idea of making this sort of thing work seamlessly,
> another possibility to consider is to use the *real* uid  of the process
> which performs the mount do authenticate SETCLIENTID - either as the primary
> authentication or as a fall-back.
> Then the a mountpoint with "users" listed as an option in /etc/fstab can be
> mounted by anyone with appropriate credentials and no credential should be
> needed for root, and not machine credential should be needed either.

Perhaps a more common use case with NFSv4 is secure automounter mounts.  In that case, the server can be configured to allow sec=sys access to it's pseudo-fs, then require Kerberos for its real file systems.

In operation, the client mounts the server's pseudo-fs without needing any GSS context.  When it transitions into a protected file system, the server returns NFS4ERR_WRONGSEC and the client should be able to negotiate up to AUTH_GSS using the user's credential.  (Mind you I haven't tried this, I'm simply hand-waving based on reasonable assumptions).

The key feature of this use case is that the client is not configured for anything but /net.  Mount options like "sec=krb5" or "users" are simply not in the picture.

The question is still whether, if the client doesn't have a keytab, lease management should use the user's credential or AUTH_SYS.  I maintain that AUTH_SYS is a better choice.  The kernel cannot possibly know a priori, without configuration, that this is a single-user client and that it is OK to use a user's Kerberos credential for lease management.

I'll stress that the reason we prefer a machine credential (either AUTH_GSS or AUTH_SYS) is because a client must present a consistent principal and flavor to the server for lease management.

A server can refuse to recognize a SETCLIENTID (for example, after a sudden client reboot) if there is already a lease on the server for using the presented nfs_client_id4.id string that was established using a different security flavor or principal.  Otherwise there would be no point in securing lease management.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com