[NFS] NFS/krb and batch jobs - doable?

[NFS] NFS/krb and batch jobs - doable?

[raini-9HxftnAiGddWk0Htik3J/w]

I've been struggling for some time to understand how to allow users of
long-running batch jobs to continue to do so in an environmnet migrated to
NFS4 with kerberos (from non-kerberized NFS), centered around the problem
of keeping their batch job credentials separate from login credentials,
which might otherwise mangle or delete them and cause the batch jobs to
die on logout - or say if the users opted for renewable tickets for the
job explicitly but the system default is non-renewable.

The central problem seems to be that NFS4 (on Linux at least) doesn't
support the concept of multiple sessions, and seems to expect to see a
standard ccache filename /tmp/krb5cc_${UID}.  Am I right in thinking this
is a fundamental limitation?  I've played with KRB5CCNAME but am led to
believe NFS ignores this; I've also added ccname template settings to
krb5.conf and PAM to try to fool NFS into separating batch job credentials
from user ones, but don't seem to be getting a robust separation.

The essential problem is that I need an environment where a user can
obtain credentials that may be different from the desirable interactive
default (e.g. longer duration, renewable), then run a batch job, ideally
in the heterogeneous ways they do currently - run via a shell, kicked off
on another machine via ssh, via condor - and not have its credentials at
risk if the user then logs into the machine it's running on, submits a
second job etc.

Is this doable currently?  I'm surprised (so far) to not run across people
who are doing this, and would be very surprised if it's not a requirement
for many.  If not, is it likely to be doable soon and what does it depend
on?

I can supply more technical details of what I've tried if useful.  Please
also let me know if there's a more appropriate place I can ask these
questions.  Thanks.



------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] NFS/krb and batch jobs - doable?

[Jeff Layton]

On Fri, 9 Oct 2009 08:15:12 -0700
raini-9HxftnAiGddWk0Htik3J/w@public.gmane.org wrote:

> I've been struggling for some time to understand how to allow users of
> long-running batch jobs to continue to do so in an environmnet migrated to
> NFS4 with kerberos (from non-kerberized NFS), centered around the problem
> of keeping their batch job credentials separate from login credentials,
> which might otherwise mangle or delete them and cause the batch jobs to
> die on logout - or say if the users opted for renewable tickets for the
> job explicitly but the system default is non-renewable.
> 
> The central problem seems to be that NFS4 (on Linux at least) doesn't
> support the concept of multiple sessions, and seems to expect to see a
> standard ccache filename /tmp/krb5cc_${UID}.  Am I right in thinking this
> is a fundamental limitation? 

No, gssd (the client side daemon) will search /tmp for anything that
looks like a credcache for the right user, verify that it is a
credcache and then pick the one with the latest TGT expiration.

>  I've played with KRB5CCNAME but am led to
> believe NFS ignores this; I've also added ccname template settings to
> krb5.conf and PAM to try to fool NFS into separating batch job credentials
> from user ones, but don't seem to be getting a robust separation.
> 

You're correct that NFS ignores $KRB5CCNAME. It uses the above (less
than optimal) heuristic instead.

> The essential problem is that I need an environment where a user can
> obtain credentials that may be different from the desirable interactive
> default (e.g. longer duration, renewable), then run a batch job, ideally
> in the heterogeneous ways they do currently - run via a shell, kicked off
> on another machine via ssh, via condor - and not have its credentials at
> risk if the user then logs into the machine it's running on, submits a
> second job etc.
> 
> Is this doable currently?  I'm surprised (so far) to not run across people
> who are doing this, and would be very surprised if it's not a requirement
> for many.  If not, is it likely to be doable soon and what does it depend
> on?
> 

Probably doable, but not trivial. IIRC, the kernel tracks credentials
by uid. You'd need to determine some way to split that up so that each
"session" has separate credentials. Once you do that, you'll have to
have the kernel pass enough info to the upcall for it to determine what
credcache it should use and modify gssd to use the new info accordingly.

-- 
Jeff Layton <jlayton@redhat.com>

Re: [NFS] NFS/krb and batch jobs - doable?

[raini-9HxftnAiGddWk0Htik3J/w]

> No, gssd (the client side daemon) will search /tmp for anything that
> looks like a credcache for the right user, verify that it is a
> credcache and then pick the one with the latest TGT expiration.

> You're correct that NFS ignores $KRB5CCNAME. It uses the above (less
> than optimal) heuristic instead.

Thanks for explaining this Jeff - this does accord with what I see - which
of course leaves my batch job system unpredictable.

> Probably doable, but not trivial. IIRC, the kernel tracks credentials
> by uid. You'd need to determine some way to split that up so that each
> "session" has separate credentials. Once you do that, you'll have to
> have the kernel pass enough info to the upcall for it to determine what
> credcache it should use and modify gssd to use the new info accordingly.

Just to be clear - you mean doable to a coder who might like to improve on
gssd/kernel credential separation, rather than a non-coding sysadmin who
needs with work within the current NFS/gssd framework?

Re: [NFS] NFS/krb and batch jobs - doable?

[raini-9HxftnAiGddWk0Htik3J/w]

>> No, gssd (the client side daemon) will search /tmp for anything that
>> looks like a credcache for the right user, verify that it is a
>> credcache and then pick the one with the latest TGT expiration.
>
>> You're correct that NFS ignores $KRB5CCNAME. It uses the above (less
>> than optimal) heuristic instead.
>
> Thanks for explaining this Jeff - this does accord with what I see - which
> of course leaves my batch job system unpredictable.
>
>> Probably doable, but not trivial. IIRC, the kernel tracks credentials
>> by uid. You'd need to determine some way to split that up so that each
>> "session" has separate credentials. Once you do that, you'll have to
>> have the kernel pass enough info to the upcall for it to determine what
>> credcache it should use and modify gssd to use the new info accordingly.
>
> Just to be clear - you mean doable to a coder who might like to improve on
> gssd/kernel credential separation, rather than a non-coding sysadmin who
> needs with work within the current NFS/gssd framework?
>



------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] NFS/krb and batch jobs - doable?

[Jeff Layton]

On Fri, 9 Oct 2009 09:53:51 -0700
raini-9HxftnAiGddWk0Htik3J/w@public.gmane.org wrote:

> > No, gssd (the client side daemon) will search /tmp for anything that
> > looks like a credcache for the right user, verify that it is a
> > credcache and then pick the one with the latest TGT expiration.
> 
> > You're correct that NFS ignores $KRB5CCNAME. It uses the above (less
> > than optimal) heuristic instead.
> 
> Thanks for explaining this Jeff - this does accord with what I see - which
> of course leaves my batch job system unpredictable.
> 
> > Probably doable, but not trivial. IIRC, the kernel tracks credentials
> > by uid. You'd need to determine some way to split that up so that each
> > "session" has separate credentials. Once you do that, you'll have to
> > have the kernel pass enough info to the upcall for it to determine what
> > credcache it should use and modify gssd to use the new info accordingly.
> 
> Just to be clear - you mean doable to a coder who might like to improve on
> gssd/kernel credential separation, rather than a non-coding sysadmin who
> needs with work within the current NFS/gssd framework?
> 

Correct, that's what I mean. It'll mean modifying kernel and rpc.gssd
code.

-- 
Jeff Layton <jlayton@redhat.com>

Re: [NFS] NFS/krb and batch jobs - doable?

[raini-9HxftnAiGddWk0Htik3J/w]

Jeff Layton <jlayton@redhat.com> said:
>> Just to be clear - you mean doable to a coder who might like to improve
>> on
>> gssd/kernel credential separation, rather than a non-coding sysadmin who
>> needs with work within the current NFS/gssd framework?
>>
>
> Correct, that's what I mean. It'll mean modifying kernel and rpc.gssd
> code.

Thanks for confirming.  Skipping back a little:

>> > No, gssd (the client side daemon) will search /tmp for anything that
>> > looks like a credcache for the right user, verify that it is a
>> > credcache and then pick the one with the latest TGT expiration.

Kevin Coffman on the NFS4 list actually implied this used simple mtime
rather than actually scanning /tmp/krb5cc_uid* for ccache files with the
latest TGT expiration, which is how I originally read your statement. 
This seemingly would make a difference in an environment with a batch job
with a long lifetime ticket and subsequent interactive login generating a
separate ccache file with a shorter lifetime but newer mtime.

I'm not a coder but I scanned krb5_util.c in the gssd code, and it *seems*
to me it only looks at mtime, although what you suggest would be more
optimal.  Could you confirm whether it's scanning ccache files for longest
TGT, or just using mtime?

Re: [NFS] NFS/krb and batch jobs - doable?

[Jeff Layton]

On Tue, 13 Oct 2009 08:28:52 -0700
raini-9HxftnAiGddWk0Htik3J/w@public.gmane.org wrote:

> Jeff Layton <jlayton@redhat.com> said:
> >> Just to be clear - you mean doable to a coder who might like to improve
> >> on
> >> gssd/kernel credential separation, rather than a non-coding sysadmin who
> >> needs with work within the current NFS/gssd framework?
> >>
> >
> > Correct, that's what I mean. It'll mean modifying kernel and rpc.gssd
> > code.
> 
> Thanks for confirming.  Skipping back a little:
> 
> >> > No, gssd (the client side daemon) will search /tmp for anything that
> >> > looks like a credcache for the right user, verify that it is a
> >> > credcache and then pick the one with the latest TGT expiration.
> 
> Kevin Coffman on the NFS4 list actually implied this used simple mtime
> rather than actually scanning /tmp/krb5cc_uid* for ccache files with the
> latest TGT expiration, which is how I originally read your statement. 
> This seemingly would make a difference in an environment with a batch job
> with a long lifetime ticket and subsequent interactive login generating a
> separate ccache file with a shorter lifetime but newer mtime.
> 
> I'm not a coder but I scanned krb5_util.c in the gssd code, and it *seems*
> to me it only looks at mtime, although what you suggest would be more
> optimal.  Could you confirm whether it's scanning ccache files for longest
> TGT, or just using mtime?
> 

You and Kevin are correct. rpc.gssd only looks at the mtime. When I did
the work to allow the CIFS SPNGEO upcall to find alternate credcaches,
I implemented the behavior I described (prefer the latest TGT
expiration) -- sorry for the confusion...

It probably wouldn't be too hard to change rpc.gssd to prefer
credcaches with the latest TGT expiration if it was considered a
desirable change.

Kevin, any thoughts?

-- 
Jeff Layton <jlayton@redhat.com>

Re: [NFS] NFS/krb and batch jobs - doable?

[Kevin Coffman]

On Tue, Oct 13, 2009 at 11:44 AM, Jeff Layton <jlayton@redhat.com> wrot=
e:
> On Tue, 13 Oct 2009 08:28:52 -0700
> raini-9HxftnAiGddWk0Htik3J/w@public.gmane.org wrote:
>
>> Jeff Layton <jlayton@redhat.com> said:
>> >> Just to be clear - you mean doable to a coder who might like to i=
mprove
>> >> on
>> >> gssd/kernel credential separation, rather than a non-coding sysad=
min who
>> >> needs with work within the current NFS/gssd framework?
>> >>
>> >
>> > Correct, that's what I mean. It'll mean modifying kernel and rpc.g=
ssd
>> > code.
>>
>> Thanks for confirming. =A0Skipping back a little:
>>
>> >> > No, gssd (the client side daemon) will search /tmp for anything=
 that
>> >> > looks like a credcache for the right user, verify that it is a
>> >> > credcache and then pick the one with the latest TGT expiration.
>>
>> Kevin Coffman on the NFS4 list actually implied this used simple mti=
me
>> rather than actually scanning /tmp/krb5cc_uid* for ccache files with=
 the
>> latest TGT expiration, which is how I originally read your statement=
=2E
>> This seemingly would make a difference in an environment with a batc=
h job
>> with a long lifetime ticket and subsequent interactive login generat=
ing a
>> separate ccache file with a shorter lifetime but newer mtime.
>>
>> I'm not a coder but I scanned krb5_util.c in the gssd code, and it *=
seems*
>> to me it only looks at mtime, although what you suggest would be mor=
e
>> optimal. =A0Could you confirm whether it's scanning ccache files for=
 longest
>> TGT, or just using mtime?
>>
>
> You and Kevin are correct. rpc.gssd only looks at the mtime. When I d=
id
> the work to allow the CIFS SPNGEO upcall to find alternate credcaches=
,
> I implemented the behavior I described (prefer the latest TGT
> expiration) -- sorry for the confusion...
>
> It probably wouldn't be too hard to change rpc.gssd to prefer
> credcaches with the latest TGT expiration if it was considered a
> desirable change.
>
> Kevin, any thoughts?

I agree it shouldn't be too hard to change if that behavior is desirabl=
e/useful.

Re: [NFS] NFS/krb and batch jobs - doable?

[raini]

> You and Kevin are correct. rpc.gssd only looks at the mtime. When I did
> the work to allow the CIFS SPNGEO upcall to find alternate credcaches,
> I implemented the behavior I described (prefer the latest TGT
> expiration) -- sorry for the confusion...
>
> It probably wouldn't be too hard to change rpc.gssd to prefer
> credcaches with the latest TGT expiration if it was considered a
> desirable change.
>
> Kevin, any thoughts?

This would be a big plus from me - I still wouldn't be able to create
per-job ccaches of course, but if a user who knew they needed to run a job
could create a long lifetime renewable ticket in /tmp/krb5cc_<uid>_batch,
say, and NFS would use this in preference to a later login ticket, this
would really help.

Re: [NFS] NFS/krb and batch jobs - doable?

[Trond Myklebust]

On Tue, 2009-10-13 at 11:51 -0400, Kevin Coffman wrote:
> On Tue, Oct 13, 2009 at 11:44 AM, Jeff Layton <jlayton@redhat.com> wrote:
> > On Tue, 13 Oct 2009 08:28:52 -0700
> > raini@rainiday.com wrote:
> >
> >> Jeff Layton <jlayton@redhat.com> said:
> >> >> Just to be clear - you mean doable to a coder who might like to improve
> >> >> on
> >> >> gssd/kernel credential separation, rather than a non-coding sysadmin who
> >> >> needs with work within the current NFS/gssd framework?
> >> >>
> >> >
> >> > Correct, that's what I mean. It'll mean modifying kernel and rpc.gssd
> >> > code.
> >>
> >> Thanks for confirming.  Skipping back a little:
> >>
> >> >> > No, gssd (the client side daemon) will search /tmp for anything that
> >> >> > looks like a credcache for the right user, verify that it is a
> >> >> > credcache and then pick the one with the latest TGT expiration.
> >>
> >> Kevin Coffman on the NFS4 list actually implied this used simple mtime
> >> rather than actually scanning /tmp/krb5cc_uid* for ccache files with the
> >> latest TGT expiration, which is how I originally read your statement.
> >> This seemingly would make a difference in an environment with a batch job
> >> with a long lifetime ticket and subsequent interactive login generating a
> >> separate ccache file with a shorter lifetime but newer mtime.
> >>
> >> I'm not a coder but I scanned krb5_util.c in the gssd code, and it *seems*
> >> to me it only looks at mtime, although what you suggest would be more
> >> optimal.  Could you confirm whether it's scanning ccache files for longest
> >> TGT, or just using mtime?
> >>
> >
> > You and Kevin are correct. rpc.gssd only looks at the mtime. When I did
> > the work to allow the CIFS SPNGEO upcall to find alternate credcaches,
> > I implemented the behavior I described (prefer the latest TGT
> > expiration) -- sorry for the confusion...
> >
> > It probably wouldn't be too hard to change rpc.gssd to prefer
> > credcaches with the latest TGT expiration if it was considered a
> > desirable change.
> >
> > Kevin, any thoughts?
> 
> I agree it shouldn't be too hard to change if that behavior is desirable/useful.

Shouldn't rpc.gssd in any case be looking for all _valid_ credcaches,
rather than just any old credcache-like file?

If you stick to that principle, then if there is one file that contains
a still-unexpired TGT, and one with a TGT that has expired, then the
first file should always be preferred, irrespective of the value of the
mtimes...

Cheers
  Trond

Re: [NFS] NFS/krb and batch jobs - doable?

[Jeff Layton]

On Tue, 13 Oct 2009 12:56:20 -0400
Trond Myklebust <trond.myklebust@fys.uio.no> wrote:

> On Tue, 2009-10-13 at 11:51 -0400, Kevin Coffman wrote:
> > On Tue, Oct 13, 2009 at 11:44 AM, Jeff Layton <jlayton@redhat.com> wrote:
> > > On Tue, 13 Oct 2009 08:28:52 -0700
> > > raini@rainiday.com wrote:
> > >
> > >> Jeff Layton <jlayton@redhat.com> said:
> > >> >> Just to be clear - you mean doable to a coder who might like to improve
> > >> >> on
> > >> >> gssd/kernel credential separation, rather than a non-coding sysadmin who
> > >> >> needs with work within the current NFS/gssd framework?
> > >> >>
> > >> >
> > >> > Correct, that's what I mean. It'll mean modifying kernel and rpc.gssd
> > >> > code.
> > >>
> > >> Thanks for confirming.  Skipping back a little:
> > >>
> > >> >> > No, gssd (the client side daemon) will search /tmp for anything that
> > >> >> > looks like a credcache for the right user, verify that it is a
> > >> >> > credcache and then pick the one with the latest TGT expiration.
> > >>
> > >> Kevin Coffman on the NFS4 list actually implied this used simple mtime
> > >> rather than actually scanning /tmp/krb5cc_uid* for ccache files with the
> > >> latest TGT expiration, which is how I originally read your statement.
> > >> This seemingly would make a difference in an environment with a batch job
> > >> with a long lifetime ticket and subsequent interactive login generating a
> > >> separate ccache file with a shorter lifetime but newer mtime.
> > >>
> > >> I'm not a coder but I scanned krb5_util.c in the gssd code, and it *seems*
> > >> to me it only looks at mtime, although what you suggest would be more
> > >> optimal.  Could you confirm whether it's scanning ccache files for longest
> > >> TGT, or just using mtime?
> > >>
> > >
> > > You and Kevin are correct. rpc.gssd only looks at the mtime. When I did
> > > the work to allow the CIFS SPNGEO upcall to find alternate credcaches,
> > > I implemented the behavior I described (prefer the latest TGT
> > > expiration) -- sorry for the confusion...
> > >
> > > It probably wouldn't be too hard to change rpc.gssd to prefer
> > > credcaches with the latest TGT expiration if it was considered a
> > > desirable change.
> > >
> > > Kevin, any thoughts?
> > 
> > I agree it shouldn't be too hard to change if that behavior is desirable/useful.
> 
> Shouldn't rpc.gssd in any case be looking for all _valid_ credcaches,
> rather than just any old credcache-like file?
> 
> If you stick to that principle, then if there is one file that contains
> a still-unexpired TGT, and one with a TGT that has expired, then the
> first file should always be preferred, irrespective of the value of the
> mtimes...
> 

Correct...and gssd actually does check the validity of the cache. If
TGT has expired or it's not valid for some other reason, then it skips
it and moves on.

The problem comes when you have more than one valid credcache. In that
case it picks the one with the latest mtime. It seems that it should
instead pick the one with the latest TGT expiration time.

-- 
Jeff Layton <jlayton@redhat.com>

Re: [NFS] NFS/krb and batch jobs - doable?

[Jeff Layton]

[-- Attachment #1: Type: text/plain, Size: 1080 bytes --]

On Tue, 13 Oct 2009 08:59:29 -0700
raini@rainiday.com wrote:

> 
> > You and Kevin are correct. rpc.gssd only looks at the mtime. When I did
> > the work to allow the CIFS SPNGEO upcall to find alternate credcaches,
> > I implemented the behavior I described (prefer the latest TGT
> > expiration) -- sorry for the confusion...
> >
> > It probably wouldn't be too hard to change rpc.gssd to prefer
> > credcaches with the latest TGT expiration if it was considered a
> > desirable change.
> >
> > Kevin, any thoughts?
> 
> This would be a big plus from me - I still wouldn't be able to create
> per-job ccaches of course, but if a user who knew they needed to run a job
> could create a long lifetime renewable ticket in /tmp/krb5cc_<uid>_batch,
> say, and NFS would use this in preference to a later login ticket, this
> would really help.
> 
> 

Ok, here's a proposed patch...only compile-tested so far. I don't have
time at the moment to test it more extensively so if you could test it
out and report back, that would be helpful.

Thanks,
-- 
Jeff Layton <jlayton@redhat.com>

[-- Attachment #2: 0001-gssd-prefer-credcaches-with-latest-TGT-expiration.patch --]
[-- Type: text/x-patch, Size: 5813 bytes --]

>From 39564369e8a22b97f4a3021eb527eaf6524ca557 Mon Sep 17 00:00:00 2001
From: Jeff Layton <jlayton@redhat.com>
Date: Tue, 13 Oct 2009 13:27:52 -0400
Subject: [PATCH] gssd: prefer credcaches with latest TGT expiration

(Note: this is only compile-tested so far)

gssd currently walks all of the entities in the credcache dir that look
like credcaches, verifies that they are valid and picks the one with the
latest mtime.

There's a problem here however, the one with the latest mtime might still
expire very soon after we pick it. Instead of doing that, have it instead
pick the credcache that has the latest TGT expiration.

With this, someone who wants to run a long-running batch job can get a
TGT with a long expiration time and have some reasonable hope that it'll
run to completion.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
---
 utils/gssd/krb5_util.c |   72 ++++++++++++++++++-----------------------------
 1 files changed, 28 insertions(+), 44 deletions(-)

diff --git a/utils/gssd/krb5_util.c b/utils/gssd/krb5_util.c
index 78e9775..8f1f56d 100644
--- a/utils/gssd/krb5_util.c
+++ b/utils/gssd/krb5_util.c
@@ -134,11 +134,11 @@ struct gssd_k5_kt_princ *gssd_k5_kt_princ_list = NULL;
 /*==========================*/
 
 static int select_krb5_ccache(const struct dirent *d);
-static int gssd_find_existing_krb5_ccache(uid_t uid, char *dirname,
+static time_t gssd_find_existing_krb5_ccache(uid_t uid, char *dirname,
 		struct dirent **d);
 static int gssd_get_single_krb5_cred(krb5_context context,
 		krb5_keytab kt, struct gssd_k5_kt_princ *ple, int nocache);
-static int query_krb5_ccache(const char* cred_cache, char **ret_princname,
+static time_t query_krb5_ccache(const char* cred_cache, char **ret_princname,
 		char **ret_realm);
 
 /*
@@ -171,24 +171,22 @@ select_krb5_ccache(const struct dirent *d)
  * The caller is responsible for freeing the dirent if one is returned.
  *
  * Returns:
- *	0 => could not find an existing entry
- *	1 => found an existing entry
+ *	 0 => could not find an existing entry
+ *	>0 => found an existing entry
  */
-static int
+static time_t
 gssd_find_existing_krb5_ccache(uid_t uid, char *dirname, struct dirent **d)
 {
 	struct dirent **namelist;
 	int n;
 	int i;
-	int found = 0;
 	struct dirent *best_match_dir = NULL;
-	struct stat best_match_stat, tmp_stat;
+	struct stat tmp_stat;
 	char buf[1030];
 	char *princname = NULL;
 	char *realm = NULL;
-	int score, best_match_score = 0;
+	time_t tmp_time, best_match_time = 0;
 
-	memset(&best_match_stat, 0, sizeof(best_match_stat));
 	*d = NULL;
 	n = scandir(dirname, &namelist, select_krb5_ccache, 0);
 	if (n < 0) {
@@ -225,18 +223,15 @@ gssd_find_existing_krb5_ccache(uid_t uid, char *dirname, struct dirent **d)
 				free(namelist[i]);
 				continue;
 			}
-			if (!query_krb5_ccache(buf, &princname, &realm)) {
+
+			tmp_time = query_krb5_ccache(buf, &princname, &realm);
+			if (!tmp_time) {
 				printerr(3, "CC file '%s' is expired or corrupt\n",
 					 statname);
 				free(namelist[i]);
 				continue;
 			}
 
-			score = 0;
-			if (preferred_realm &&
-					strcmp(realm, preferred_realm) == 0) 
-				score++;
-
 			printerr(3, "CC file '%s'(%s@%s) passed all checks and"
 				    " has mtime of %u\n",
 				 statname, princname, realm, 
@@ -246,49 +241,38 @@ gssd_find_existing_krb5_ccache(uid_t uid, char *dirname, struct dirent **d)
 			 * recent (the one with the latest mtime), and
 			 * don't free the dirent
 			 */
-			if (!found) {
+			if (!best_match_time) {
 				best_match_dir = namelist[i];
-				best_match_stat = tmp_stat;
-				best_match_score = score;
-				found++;
-			}
-			else {
+				best_match_time = tmp_time;
+			} else {
 				/*
-				 * If current score is higher than best match 
-				 * score, we use the current match. Otherwise,
-				 * if the current match has an mtime later
-				 * than the one we are looking at, then use
-				 * the current match.  Otherwise, we still
-				 * have the best match.
+				 * prefer the credcache that has the latest
+				 * TGT expiration time. This is helpful when
+				 * for ensuring that long-running jobs don't
+				 * lose their credentials.
 				 */
-				if (best_match_score < score ||
-				    (best_match_score == score && 
-				       tmp_stat.st_mtime >
-					    best_match_stat.st_mtime)) {
+				if (tmp_time > best_match_time) {
 					free(best_match_dir);
 					best_match_dir = namelist[i];
-					best_match_stat = tmp_stat;
-					best_match_score = score;
-				}
-				else {
+					best_match_time = tmp_time;
+				} else {
 					free(namelist[i]);
 				}
 				printerr(3, "CC file '%s/%s' is our "
 					    "current best match "
-					    "with mtime of %u\n",
+					    "with expire time of %u\n",
 					 dirname, best_match_dir->d_name,
-					 best_match_stat.st_mtime);
+					 best_match_time);
 			}
 			free(princname);
 			free(realm);
 		}
 		free(namelist);
 	}
-	if (found)
-	{
+	if (best_match_time)
 		*d = best_match_dir;
-	}
-	return found;
+
+	return best_match_time;
 }
 
 
@@ -935,7 +919,7 @@ static inline int data_is_equal(krb5_data d1, krb5_data d2)
 		&& memcmp(d1.data, d2.data, d1.length) == 0);
 }
 
-static int
+static time_t
 check_for_tgt(krb5_context context, krb5_ccache ccache,
 	      krb5_principal principal)
 {
@@ -959,7 +943,7 @@ check_for_tgt(krb5_context context, krb5_ccache ccache,
 				data_is_equal(creds.server->data[1],
 					      principal->realm) &&
 				creds.times.endtime > time(NULL))
-			found = 1;
+			found = creds.times.endtime;
 		krb5_free_cred_contents(context, &creds);
 	}
 	krb5_cc_end_seq_get(context, ccache, &cur);
@@ -967,7 +951,7 @@ check_for_tgt(krb5_context context, krb5_ccache ccache,
 	return found;
 }
 
-static int
+static time_t
 query_krb5_ccache(const char* cred_cache, char **ret_princname,
 		  char **ret_realm)
 {
-- 
1.6.0.6

Re: [NFS] NFS/krb and batch jobs - doable?

[Trond Myklebust]

On Tue, 2009-10-13 at 13:27 -0400, Jeff Layton wrote:
> Correct...and gssd actually does check the validity of the cache. If
> TGT has expired or it's not valid for some other reason, then it skips
> it and moves on.
> 
> The problem comes when you have more than one valid credcache. In that
> case it picks the one with the latest mtime. It seems that it should
> instead pick the one with the latest TGT expiration time.

So why do you think that is a problem? The result should be that
rpc.gssd always ends up with a valid credential as long as there is at
least one with a valid TGT.
IOW: Who cares if the GSS session isn't going to last as long, as long
as the RPC client can always instantiate a new one.

Trond

Re: [NFS] NFS/krb and batch jobs - doable?

[Jeff Layton]

On Tue, 13 Oct 2009 13:31:38 -0400
Jeff Layton <jlayton@redhat.com> wrote:

> On Tue, 13 Oct 2009 08:59:29 -0700
> raini@rainiday.com wrote:
> 
> > 
> > > You and Kevin are correct. rpc.gssd only looks at the mtime. When I did
> > > the work to allow the CIFS SPNGEO upcall to find alternate credcaches,
> > > I implemented the behavior I described (prefer the latest TGT
> > > expiration) -- sorry for the confusion...
> > >
> > > It probably wouldn't be too hard to change rpc.gssd to prefer
> > > credcaches with the latest TGT expiration if it was considered a
> > > desirable change.
> > >
> > > Kevin, any thoughts?
> > 
> > This would be a big plus from me - I still wouldn't be able to create
> > per-job ccaches of course, but if a user who knew they needed to run a job
> > could create a long lifetime renewable ticket in /tmp/krb5cc_<uid>_batch,
> > say, and NFS would use this in preference to a later login ticket, this
> > would really help.
> > 
> > 
> 
> Ok, here's a proposed patch...only compile-tested so far. I don't have
> time at the moment to test it more extensively so if you could test it
> out and report back, that would be helpful.
> 

Looks like this patch will probably break the "preferred realm" code.
It'll have to be respun to fix that, but it should work as expected in
a single-realm environment.

-- 
Jeff Layton <jlayton@redhat.com>

Re: [NFS] NFS/krb and batch jobs - doable?

[Jeff Layton]

On Tue, 13 Oct 2009 13:51:33 -0400
Trond Myklebust <trond.myklebust@fys.uio.no> wrote:

> On Tue, 2009-10-13 at 13:27 -0400, Jeff Layton wrote:
> > Correct...and gssd actually does check the validity of the cache. If
> > TGT has expired or it's not valid for some other reason, then it skips
> > it and moves on.
> > 
> > The problem comes when you have more than one valid credcache. In that
> > case it picks the one with the latest mtime. It seems that it should
> > instead pick the one with the latest TGT expiration time.
> 
> So why do you think that is a problem? The result should be that
> rpc.gssd always ends up with a valid credential as long as there is at
> least one with a valid TGT.
> IOW: Who cares if the GSS session isn't going to last as long, as long
> as the RPC client can always instantiate a new one.
> 

Hrm...good point. I suppose that as long as gssd can pick a new
credcache if the context expires then this patch is superfluous. Wasn't
that support only added fairly recently (around a year ago?)? If so, it
may just be that raini isn't using a recent enough nfs-utils...

-- 
Jeff Layton <jlayton@redhat.com>

Re: [NFS] NFS/krb and batch jobs - doable?

[raini]

> On Tue, 13 Oct 2009 13:51:33 -0400
> Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
>> On Tue, 2009-10-13 at 13:27 -0400, Jeff Layton wrote:
>> > Correct...and gssd actually does check the validity of the cache. If
>> > TGT has expired or it's not valid for some other reason, then it skips
>> > it and moves on.
>> >
>> > The problem comes when you have more than one valid credcache. In that
>> > case it picks the one with the latest mtime. It seems that it should
>> > instead pick the one with the latest TGT expiration time.
>>
>> So why do you think that is a problem? The result should be that
>> rpc.gssd always ends up with a valid credential as long as there is at
>> least one with a valid TGT.
>> IOW: Who cares if the GSS session isn't going to last as long, as long
>> as the RPC client can always instantiate a new one.
>>
>
> Hrm...good point. I suppose that as long as gssd can pick a new
> credcache if the context expires then this patch is superfluous. Wasn't
> that support only added fairly recently (around a year ago?)? If so, it
> may just be that raini isn't using a recent enough nfs-utils...

Hm - well I'm stuck on production machines (RHEL5) so currently on
nfs-utils 1.0.9 which I'm going to take a wild guess may be problematic
either way.  Could someone point me to information on this change (I see
little in http://www.kernel.org/pub/linux/utils/nfs/)?

The reason I thought the new code would be useful is that if default
tickets are non-renewable and short lifetime, it seems sensible for gssd
to spot and use a longer lifetime renewable ticket in another ccache file
- and say use krenew to keep the job alive (or even cope with the user
renewing the ticket manually).

Seems to me therefore that in the absence of per-session ccaches, gssd
should prefer long lifetime, and renewable.

Would the newer code you mention cope with this situation already?

Re: [NFS] NFS/krb and batch jobs - doable?

[raini]

> On Tue, 13 Oct 2009 08:59:29 -0700
> raini@rainiday.com wrote:
>
>> > You and Kevin are correct. rpc.gssd only looks at the mtime. When I
>> did
>> > the work to allow the CIFS SPNGEO upcall to find alternate credcaches,
>> > I implemented the behavior I described (prefer the latest TGT
>> > expiration) -- sorry for the confusion...
>> >
>> > It probably wouldn't be too hard to change rpc.gssd to prefer
>> > credcaches with the latest TGT expiration if it was considered a
>> > desirable change.
>> >
>> > Kevin, any thoughts?
>>
>> This would be a big plus from me - I still wouldn't be able to create
>> per-job ccaches of course, but if a user who knew they needed to run a
>> job
>> could create a long lifetime renewable ticket in
>> /tmp/krb5cc_<uid>_batch,
>> say, and NFS would use this in preference to a later login ticket, this
>> would really help.
>
> Ok, here's a proposed patch...only compile-tested so far. I don't have
> time at the moment to test it more extensively so if you could test it
> out and report back, that would be helpful.

Thanks Jeff - this looks extremely useful, caveat my other comment (and
perhaps lack of understanding) on the list today about what's happened in
recent nfs-utils which I'd like to clarify.

I may have trouble testing this in the short term as I'm largely bound to
production environments - but will get to back to you if I can.

Re: [NFS] NFS/krb and batch jobs - doable?

[Trond Myklebust]

On Wed, 2009-10-14 at 09:47 -0700, raini@rainiday.com wrote:
> > On Tue, 13 Oct 2009 13:51:33 -0400
> > Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> >
> >> On Tue, 2009-10-13 at 13:27 -0400, Jeff Layton wrote:
> >> > Correct...and gssd actually does check the validity of the cache. If
> >> > TGT has expired or it's not valid for some other reason, then it skips
> >> > it and moves on.
> >> >
> >> > The problem comes when you have more than one valid credcache. In that
> >> > case it picks the one with the latest mtime. It seems that it should
> >> > instead pick the one with the latest TGT expiration time.
> >>
> >> So why do you think that is a problem? The result should be that
> >> rpc.gssd always ends up with a valid credential as long as there is at
> >> least one with a valid TGT.
> >> IOW: Who cares if the GSS session isn't going to last as long, as long
> >> as the RPC client can always instantiate a new one.
> >>
> >
> > Hrm...good point. I suppose that as long as gssd can pick a new
> > credcache if the context expires then this patch is superfluous. Wasn't
> > that support only added fairly recently (around a year ago?)? If so, it
> > may just be that raini isn't using a recent enough nfs-utils...
> 
> Hm - well I'm stuck on production machines (RHEL5) so currently on
> nfs-utils 1.0.9 which I'm going to take a wild guess may be problematic
> either way.  Could someone point me to information on this change (I see
> little in http://www.kernel.org/pub/linux/utils/nfs/)?
> 
> The reason I thought the new code would be useful is that if default
> tickets are non-renewable and short lifetime, it seems sensible for gssd
> to spot and use a longer lifetime renewable ticket in another ccache file
> - and say use krenew to keep the job alive (or even cope with the user
> renewing the ticket manually).
> 
> Seems to me therefore that in the absence of per-session ccaches, gssd
> should prefer long lifetime, and renewable.

No. That is not a necessary condition.

> Would the newer code you mention cope with this situation already?

Yes. Please reread the thread carefully again.

As long as gssd always select from among the valid (i.e. unexpired)
TGTs, then the actual remaining lifetime of the particular TGT that was
selected doesn't matter; gssd will always be able to renew the resulting
RPCSEC_GSS session when it expires.

  Trond

Re: [NFS] NFS/krb and batch jobs - doable?

[Jeff Layton]

On Wed, 14 Oct 2009 10:00:59 -0700
raini@rainiday.com wrote:

> > On Tue, 13 Oct 2009 08:59:29 -0700
> > raini@rainiday.com wrote:
> >
> >> > You and Kevin are correct. rpc.gssd only looks at the mtime. When I
> >> did
> >> > the work to allow the CIFS SPNGEO upcall to find alternate credcaches,
> >> > I implemented the behavior I described (prefer the latest TGT
> >> > expiration) -- sorry for the confusion...
> >> >
> >> > It probably wouldn't be too hard to change rpc.gssd to prefer
> >> > credcaches with the latest TGT expiration if it was considered a
> >> > desirable change.
> >> >
> >> > Kevin, any thoughts?
> >>
> >> This would be a big plus from me - I still wouldn't be able to create
> >> per-job ccaches of course, but if a user who knew they needed to run a
> >> job
> >> could create a long lifetime renewable ticket in
> >> /tmp/krb5cc_<uid>_batch,
> >> say, and NFS would use this in preference to a later login ticket, this
> >> would really help.
> >
> > Ok, here's a proposed patch...only compile-tested so far. I don't have
> > time at the moment to test it more extensively so if you could test it
> > out and report back, that would be helpful.
> 
> Thanks Jeff - this looks extremely useful, caveat my other comment (and
> perhaps lack of understanding) on the list today about what's happened in
> recent nfs-utils which I'd like to clarify.
> 
> I may have trouble testing this in the short term as I'm largely bound to
> production environments - but will get to back to you if I can.
> 

Actually...I'm not convinced that it is that useful. As Trond pointed
out, when the credentials expire, the kernel should upcall for new
creds. As long as there is a valid TGT in a credcache for that user
somewhere then it should just pick up that one and keep humming along.
If that's not working for some reason then that's likely a bug.

-- 
Jeff Layton <jlayton@redhat.com>

Re: [NFS] NFS/krb and batch jobs - doable?

[Kevin Coffman]

On Wed, Oct 14, 2009 at 12:47 PM,  <raini@rainiday.com> wrote:
>> On Tue, 13 Oct 2009 13:51:33 -0400
>> Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>>
>>> On Tue, 2009-10-13 at 13:27 -0400, Jeff Layton wrote:
>>> > Correct...and gssd actually does check the validity of the cache. If
>>> > TGT has expired or it's not valid for some other reason, then it skips
>>> > it and moves on.
>>> >
>>> > The problem comes when you have more than one valid credcache. In that
>>> > case it picks the one with the latest mtime. It seems that it should
>>> > instead pick the one with the latest TGT expiration time.
>>>
>>> So why do you think that is a problem? The result should be that
>>> rpc.gssd always ends up with a valid credential as long as there is at
>>> least one with a valid TGT.
>>> IOW: Who cares if the GSS session isn't going to last as long, as long
>>> as the RPC client can always instantiate a new one.
>>>
>>
>> Hrm...good point. I suppose that as long as gssd can pick a new
>> credcache if the context expires then this patch is superfluous. Wasn't
>> that support only added fairly recently (around a year ago?)? If so, it
>> may just be that raini isn't using a recent enough nfs-utils...
>
> Hm - well I'm stuck on production machines (RHEL5) so currently on
> nfs-utils 1.0.9 which I'm going to take a wild guess may be problematic
> either way.  Could someone point me to information on this change (I see
> little in http://www.kernel.org/pub/linux/utils/nfs/)?
>
> The reason I thought the new code would be useful is that if default
> tickets are non-renewable and short lifetime, it seems sensible for gssd
> to spot and use a longer lifetime renewable ticket in another ccache file
> - and say use krenew to keep the job alive (or even cope with the user
> renewing the ticket manually).
>
> Seems to me therefore that in the absence of per-session ccaches, gssd
> should prefer long lifetime, and renewable.
>
> Would the newer code you mention cope with this situation already?

The change that adds the check for "valid" credentials caches
(including expiration) was not added until nfs-utils-1.1.3.

With that version of rpc.gssd, Jeff and Trond's descriptions are correct.

K.C.