* Strange cred expiry behavior
@ 2013-10-21 20:07 Weston Andros Adamson
2013-10-21 23:53 ` Weston Andros Adamson
0 siblings, 1 reply; 10+ messages in thread
From: Weston Andros Adamson @ 2013-10-21 20:07 UTC (permalink / raw)
To: linux-nfs list
I saw this issue when revived my branch for using SP4_MACH_CRED only in the error (EKEYEXPIRED) path, and was able to reproduce it on Trond's testing branch, so this behavior is definitely new. The client is running F19 updated pretty recently (nfs-utils 1.2.8-4.0.fc19).
The issue:
- mount a kerberos export
- kinit with a really short lifetime: "kinit -l 1m"
- run something (like dd) that will hold a file open and write to it for more than the lifetime of the kinit
! - after the credential expire, the number of keys listed in "klist" starts to grow. Notice how the "Valid starting" time is after the "Expires" time. The list seems to grow forever - my test program writes 4k every second and the list seemed to grow by 1 line per second. Example klist output:
Ticket cache: DIR::/run/user/1749600001/krb5cc/tktYmpGlX
Default principal: dros@APIKIA.FAKE
Valid starting Expires Service principal
10/21/2013 15:39:38 10/21/2013 15:40:35 krbtgt/APIKIA.FAKE@APIKIA.FAKE
10/21/2013 15:39:40 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
10/21/2013 15:40:35 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
10/21/2013 15:40:36 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
10/21/2013 15:40:37 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
10/21/2013 15:40:37 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
10/21/2013 15:40:38 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
10/21/2013 15:40:38 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
10/21/2013 15:40:39 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
10/21/2013 15:40:39 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
10/21/2013 15:40:39 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
10/21/2013 15:40:39 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
10/21/2013 15:40:40 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
10/21/2013 15:40:40 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
10/21/2013 15:40:41 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
10/21/2013 15:40:41 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
10/21/2013 15:40:42 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
10/21/2013 15:40:42 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
10/21/2013 15:40:42 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
This seems like a gssd bug in handling cred refresh upcalls. I'll debug further and hopefully find a fix.
Has anyone seen this issue before or have any thoughts on recent changes that could cause this? I definitely didn't see this a little more than a month ago when I started testing cred expiry...
-dros
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Strange cred expiry behavior
2013-10-21 20:07 Strange cred expiry behavior Weston Andros Adamson
@ 2013-10-21 23:53 ` Weston Andros Adamson
2013-10-22 2:23 ` Jeff Layton
2013-10-22 12:20 ` Simo Sorce
0 siblings, 2 replies; 10+ messages in thread
From: Weston Andros Adamson @ 2013-10-21 23:53 UTC (permalink / raw)
To: linux-nfs list; +Cc: Simo Sorce
I traced this behavior back to:
commit 302de786930a2c533068f9d8909a817b40f07c32
Author: Simo Sorce <simo@redhat.com>
Date: Fri Apr 19 13:02:36 2013 -0400
gssd: Allow GSSAPI to try to acquire credentials first.
And in particular:
- for (dirname = ccachesearch; *dirname != NULL; dirname++) {
+ /* Try first to acquire credentials directly via GSSAPI */
+ err = gssd_acquire_user_cred(uid, &gss_cred);
+ if (!err)
+ create_resp = create_auth_rpc_client(clp, &rpc_clnt, &auth, uid,
+ AUTHTYPE_KRB5, gss_cred);
+ /* if create_auth_rplc_client fails try the traditional method of
+ * trolling for credentials */
+ for (dirname = ccachesearch; create_resp != 0 && *dirname != NULL; dirname++) {
A couple of things:
- If I get rid of the "Try first to acquire credentials directly via GSSAPI" part, expiry works as before.
- It says "if create_auth_rplc_client fails try the traditional method of trolling for credentials" but I see nothing that checks to see if create_resp != 0 before falling back to the old method. I added this check and everything seems right (it doesn't fix the expiry issue).
I should have some time tomorrow to look further at the expiry issue…
-dros
On Oct 21, 2013, at 4:07 PM, Weston Andros Adamson <dros@netapp.com> wrote:
> I saw this issue when revived my branch for using SP4_MACH_CRED only in the error (EKEYEXPIRED) path, and was able to reproduce it on Trond's testing branch, so this behavior is definitely new. The client is running F19 updated pretty recently (nfs-utils 1.2.8-4.0.fc19).
>
> The issue:
>
> - mount a kerberos export
>
> - kinit with a really short lifetime: "kinit -l 1m"
>
> - run something (like dd) that will hold a file open and write to it for more than the lifetime of the kinit
>
> ! - after the credential expire, the number of keys listed in "klist" starts to grow. Notice how the "Valid starting" time is after the "Expires" time. The list seems to grow forever - my test program writes 4k every second and the list seemed to grow by 1 line per second. Example klist output:
>
>
> Ticket cache: DIR::/run/user/1749600001/krb5cc/tktYmpGlX
> Default principal: dros@APIKIA.FAKE
>
> Valid starting Expires Service principal
> 10/21/2013 15:39:38 10/21/2013 15:40:35 krbtgt/APIKIA.FAKE@APIKIA.FAKE
> 10/21/2013 15:39:40 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> 10/21/2013 15:40:35 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> 10/21/2013 15:40:36 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> 10/21/2013 15:40:37 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> 10/21/2013 15:40:37 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> 10/21/2013 15:40:38 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> 10/21/2013 15:40:38 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> 10/21/2013 15:40:39 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> 10/21/2013 15:40:39 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> 10/21/2013 15:40:39 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> 10/21/2013 15:40:39 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> 10/21/2013 15:40:40 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> 10/21/2013 15:40:40 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> 10/21/2013 15:40:41 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> 10/21/2013 15:40:41 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> 10/21/2013 15:40:42 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> 10/21/2013 15:40:42 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> 10/21/2013 15:40:42 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>
> This seems like a gssd bug in handling cred refresh upcalls. I'll debug further and hopefully find a fix.
>
> Has anyone seen this issue before or have any thoughts on recent changes that could cause this? I definitely didn't see this a little more than a month ago when I started testing cred expiry...
>
> -dros--
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Strange cred expiry behavior
2013-10-21 23:53 ` Weston Andros Adamson
@ 2013-10-22 2:23 ` Jeff Layton
2013-10-22 2:31 ` Weston Andros Adamson
2013-10-22 12:20 ` Simo Sorce
1 sibling, 1 reply; 10+ messages in thread
From: Jeff Layton @ 2013-10-22 2:23 UTC (permalink / raw)
To: Weston Andros Adamson; +Cc: linux-nfs list, Simo Sorce
On Mon, 21 Oct 2013 23:53:16 +0000
Weston Andros Adamson <dros@netapp.com> wrote:
> I traced this behavior back to:
>
> commit 302de786930a2c533068f9d8909a817b40f07c32
> Author: Simo Sorce <simo@redhat.com>
> Date: Fri Apr 19 13:02:36 2013 -0400
>
> gssd: Allow GSSAPI to try to acquire credentials first.
>
>
> And in particular:
>
> - for (dirname = ccachesearch; *dirname != NULL; dirname++) {
> + /* Try first to acquire credentials directly via GSSAPI */
> + err = gssd_acquire_user_cred(uid, &gss_cred);
> + if (!err)
> + create_resp = create_auth_rpc_client(clp, &rpc_clnt, &auth, uid,
> + AUTHTYPE_KRB5, gss_cred);
> + /* if create_auth_rplc_client fails try the traditional method of
> + * trolling for credentials */
> + for (dirname = ccachesearch; create_resp != 0 && *dirname != NULL; dirname++) {
>
> A couple of things:
>
> - If I get rid of the "Try first to acquire credentials directly via GSSAPI" part, expiry works as before.
>
Steve just merged a couple of patches from me that change this code
some. It's probably worth testing with those before you make any
changes.
> - It says "if create_auth_rplc_client fails try the traditional method of trolling for credentials" but I see nothing that checks to see if create_resp != 0 before falling back to the old method. I added this check and everything seems right (it doesn't fix the expiry issue).
>
Isn't that what the "create_resp != 0" check in the for loop does?
>
> I should have some time tomorrow to look further at the expiry issue…
>
> -dros
>
>
> On Oct 21, 2013, at 4:07 PM, Weston Andros Adamson <dros@netapp.com> wrote:
>
> > I saw this issue when revived my branch for using SP4_MACH_CRED only in the error (EKEYEXPIRED) path, and was able to reproduce it on Trond's testing branch, so this behavior is definitely new. The client is running F19 updated pretty recently (nfs-utils 1.2.8-4.0.fc19).
> >
> > The issue:
> >
> > - mount a kerberos export
> >
> > - kinit with a really short lifetime: "kinit -l 1m"
> >
> > - run something (like dd) that will hold a file open and write to it for more than the lifetime of the kinit
> >
> > ! - after the credential expire, the number of keys listed in "klist" starts to grow. Notice how the "Valid starting" time is after the "Expires" time. The list seems to grow forever - my test program writes 4k every second and the list seemed to grow by 1 line per second. Example klist output:
> >
> >
> > Ticket cache: DIR::/run/user/1749600001/krb5cc/tktYmpGlX
> > Default principal: dros@APIKIA.FAKE
> >
> > Valid starting Expires Service principal
> > 10/21/2013 15:39:38 10/21/2013 15:40:35 krbtgt/APIKIA.FAKE@APIKIA.FAKE
> > 10/21/2013 15:39:40 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> > 10/21/2013 15:40:35 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> > 10/21/2013 15:40:36 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> > 10/21/2013 15:40:37 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> > 10/21/2013 15:40:37 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> > 10/21/2013 15:40:38 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> > 10/21/2013 15:40:38 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> > 10/21/2013 15:40:39 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> > 10/21/2013 15:40:39 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> > 10/21/2013 15:40:39 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> > 10/21/2013 15:40:39 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> > 10/21/2013 15:40:40 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> > 10/21/2013 15:40:40 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> > 10/21/2013 15:40:41 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> > 10/21/2013 15:40:41 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> > 10/21/2013 15:40:42 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> > 10/21/2013 15:40:42 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> > 10/21/2013 15:40:42 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
> >
> > This seems like a gssd bug in handling cred refresh upcalls. I'll debug further and hopefully find a fix.
> >
> > Has anyone seen this issue before or have any thoughts on recent changes that could cause this? I definitely didn't see this a little more than a month ago when I started testing cred expiry...
> >
> > -dros--
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Jeff Layton <jlayton@poochiereds.net>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Strange cred expiry behavior
2013-10-22 2:23 ` Jeff Layton
@ 2013-10-22 2:31 ` Weston Andros Adamson
2013-10-22 2:38 ` Weston Andros Adamson
0 siblings, 1 reply; 10+ messages in thread
From: Weston Andros Adamson @ 2013-10-22 2:31 UTC (permalink / raw)
To: Jeff Layton; +Cc: linux-nfs list, Simo Sorce
On Oct 21, 2013, at 10:23 PM, Jeff Layton <jlayton@poochiereds.net> wrote:
> On Mon, 21 Oct 2013 23:53:16 +0000
> Weston Andros Adamson <dros@netapp.com> wrote:
>
>> I traced this behavior back to:
>>
>> commit 302de786930a2c533068f9d8909a817b40f07c32
>> Author: Simo Sorce <simo@redhat.com>
>> Date: Fri Apr 19 13:02:36 2013 -0400
>>
>> gssd: Allow GSSAPI to try to acquire credentials first.
>>
>>
>> And in particular:
>>
>> - for (dirname = ccachesearch; *dirname != NULL; dirname++) {
>> + /* Try first to acquire credentials directly via GSSAPI */
>> + err = gssd_acquire_user_cred(uid, &gss_cred);
>> + if (!err)
>> + create_resp = create_auth_rpc_client(clp, &rpc_clnt, &auth, uid,
>> + AUTHTYPE_KRB5, gss_cred);
>> + /* if create_auth_rplc_client fails try the traditional method of
>> + * trolling for credentials */
>> + for (dirname = ccachesearch; create_resp != 0 && *dirname != NULL; dirname++) {
>>
>
>
>> A couple of things:
>>
>> - If I get rid of the "Try first to acquire credentials directly via GSSAPI" part, expiry works as before.
>>
>
>
> Steve just merged a couple of patches from me that change this code
> some. It's probably worth testing with those before you make any
> changes.
>
Thanks, I'll check it out.
>> - It says "if create_auth_rplc_client fails try the traditional method of trolling for credentials" but I see nothing that checks to see if create_resp != 0 before falling back to the old method. I added this check and everything seems right (it doesn't fix the expiry issue).
>>
>
> Isn't that what the "create_resp != 0" check in the for loop does?
Heh yeah, thanks!
-dros
>
>>
>> I should have some time tomorrow to look further at the expiry issue…
>>
>> -dros
>>
>>
>
>> On Oct 21, 2013, at 4:07 PM, Weston Andros Adamson <dros@netapp.com> wrote:
>>
>>> I saw this issue when revived my branch for using SP4_MACH_CRED only in the error (EKEYEXPIRED) path, and was able to reproduce it on Trond's testing branch, so this behavior is definitely new. The client is running F19 updated pretty recently (nfs-utils 1.2.8-4.0.fc19).
>>>
>>> The issue:
>>>
>>> - mount a kerberos export
>>>
>>> - kinit with a really short lifetime: "kinit -l 1m"
>>>
>>> - run something (like dd) that will hold a file open and write to it for more than the lifetime of the kinit
>>>
>>> ! - after the credential expire, the number of keys listed in "klist" starts to grow. Notice how the "Valid starting" time is after the "Expires" time. The list seems to grow forever - my test program writes 4k every second and the list seemed to grow by 1 line per second. Example klist output:
>>>
>>>
>>> Ticket cache: DIR::/run/user/1749600001/krb5cc/tktYmpGlX
>>> Default principal: dros@APIKIA.FAKE
>>>
>>> Valid starting Expires Service principal
>>> 10/21/2013 15:39:38 10/21/2013 15:40:35 krbtgt/APIKIA.FAKE@APIKIA.FAKE
>>> 10/21/2013 15:39:40 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>> 10/21/2013 15:40:35 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>> 10/21/2013 15:40:36 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>> 10/21/2013 15:40:37 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>> 10/21/2013 15:40:37 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>> 10/21/2013 15:40:38 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>> 10/21/2013 15:40:38 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>> 10/21/2013 15:40:39 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>> 10/21/2013 15:40:39 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>> 10/21/2013 15:40:39 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>> 10/21/2013 15:40:39 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>> 10/21/2013 15:40:40 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>> 10/21/2013 15:40:40 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>> 10/21/2013 15:40:41 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>> 10/21/2013 15:40:41 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>> 10/21/2013 15:40:42 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>> 10/21/2013 15:40:42 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>> 10/21/2013 15:40:42 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>>
>>> This seems like a gssd bug in handling cred refresh upcalls. I'll debug further and hopefully find a fix.
>>>
>>> Has anyone seen this issue before or have any thoughts on recent changes that could cause this? I definitely didn't see this a little more than a month ago when I started testing cred expiry...
>>>
>>> -dros--
>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
> --
> Jeff Layton <jlayton@poochiereds.net>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Strange cred expiry behavior
2013-10-22 2:31 ` Weston Andros Adamson
@ 2013-10-22 2:38 ` Weston Andros Adamson
2013-10-22 8:48 ` Steve Dickson
2013-10-22 12:23 ` Simo Sorce
0 siblings, 2 replies; 10+ messages in thread
From: Weston Andros Adamson @ 2013-10-22 2:38 UTC (permalink / raw)
To: Jeff Layton; +Cc: linux-nfs list, Simo Sorce
On Oct 21, 2013, at 10:31 PM, Weston Andros Adamson <dros@netapp.com>
wrote:
>
> On Oct 21, 2013, at 10:23 PM, Jeff Layton <jlayton@poochiereds.net> wrote:
>
>> On Mon, 21 Oct 2013 23:53:16 +0000
>> Weston Andros Adamson <dros@netapp.com> wrote:
>>
>>> I traced this behavior back to:
>>>
>>> commit 302de786930a2c533068f9d8909a817b40f07c32
>>> Author: Simo Sorce <simo@redhat.com>
>>> Date: Fri Apr 19 13:02:36 2013 -0400
>>>
>>> gssd: Allow GSSAPI to try to acquire credentials first.
>>>
>>>
>>> And in particular:
>>>
>>> - for (dirname = ccachesearch; *dirname != NULL; dirname++) {
>>> + /* Try first to acquire credentials directly via GSSAPI */
>>> + err = gssd_acquire_user_cred(uid, &gss_cred);
>>> + if (!err)
>>> + create_resp = create_auth_rpc_client(clp, &rpc_clnt, &auth, uid,
>>> + AUTHTYPE_KRB5, gss_cred);
>>> + /* if create_auth_rplc_client fails try the traditional method of
>>> + * trolling for credentials */
>>> + for (dirname = ccachesearch; create_resp != 0 && *dirname != NULL; dirname++) {
>>>
>>
>>
>>> A couple of things:
>>>
>>> - If I get rid of the "Try first to acquire credentials directly via GSSAPI" part, expiry works as before.
>>>
>>
>>
>> Steve just merged a couple of patches from me that change this code
>> some. It's probably worth testing with those before you make any
>> changes.
>>
>
> Thanks, I'll check it out.
Bisecting brought me to 302de786930a2c533068f9d8909a817b40f07c32 and I've confirmed that the problem is still in steved's master branch as of today. Are you sure the patches you're thinking of have been merged?
-dros
>
>>> - It says "if create_auth_rplc_client fails try the traditional method of trolling for credentials" but I see nothing that checks to see if create_resp != 0 before falling back to the old method. I added this check and everything seems right (it doesn't fix the expiry issue).
>>>
>>
>> Isn't that what the "create_resp != 0" check in the for loop does?
>
> Heh yeah, thanks!
>
> -dros
>
>>
>>>
>>> I should have some time tomorrow to look further at the expiry issue…
>>>
>>> -dros
>>>
>>>
>>
>>> On Oct 21, 2013, at 4:07 PM, Weston Andros Adamson <dros@netapp.com> wrote:
>>>
>>>> I saw this issue when revived my branch for using SP4_MACH_CRED only in the error (EKEYEXPIRED) path, and was able to reproduce it on Trond's testing branch, so this behavior is definitely new. The client is running F19 updated pretty recently (nfs-utils 1.2.8-4.0.fc19).
>>>>
>>>> The issue:
>>>>
>>>> - mount a kerberos export
>>>>
>>>> - kinit with a really short lifetime: "kinit -l 1m"
>>>>
>>>> - run something (like dd) that will hold a file open and write to it for more than the lifetime of the kinit
>>>>
>>>> ! - after the credential expire, the number of keys listed in "klist" starts to grow. Notice how the "Valid starting" time is after the "Expires" time. The list seems to grow forever - my test program writes 4k every second and the list seemed to grow by 1 line per second. Example klist output:
>>>>
>>>>
>>>> Ticket cache: DIR::/run/user/1749600001/krb5cc/tktYmpGlX
>>>> Default principal: dros@APIKIA.FAKE
>>>>
>>>> Valid starting Expires Service principal
>>>> 10/21/2013 15:39:38 10/21/2013 15:40:35 krbtgt/APIKIA.FAKE@APIKIA.FAKE
>>>> 10/21/2013 15:39:40 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>>> 10/21/2013 15:40:35 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>>> 10/21/2013 15:40:36 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>>> 10/21/2013 15:40:37 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>>> 10/21/2013 15:40:37 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>>> 10/21/2013 15:40:38 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>>> 10/21/2013 15:40:38 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>>> 10/21/2013 15:40:39 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>>> 10/21/2013 15:40:39 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>>> 10/21/2013 15:40:39 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>>> 10/21/2013 15:40:39 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>>> 10/21/2013 15:40:40 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>>> 10/21/2013 15:40:40 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>>> 10/21/2013 15:40:41 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>>> 10/21/2013 15:40:41 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>>> 10/21/2013 15:40:42 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>>> 10/21/2013 15:40:42 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>>> 10/21/2013 15:40:42 10/21/2013 15:40:35 nfs/zero.apikia.fake@APIKIA.FAKE
>>>>
>>>> This seems like a gssd bug in handling cred refresh upcalls. I'll debug further and hopefully find a fix.
>>>>
>>>> Has anyone seen this issue before or have any thoughts on recent changes that could cause this? I definitely didn't see this a little more than a month ago when I started testing cred expiry...
>>>>
>>>> -dros--
>>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
>> --
>> Jeff Layton <jlayton@poochiereds.net>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Strange cred expiry behavior
2013-10-22 2:38 ` Weston Andros Adamson
@ 2013-10-22 8:48 ` Steve Dickson
2013-10-22 12:23 ` Simo Sorce
1 sibling, 0 replies; 10+ messages in thread
From: Steve Dickson @ 2013-10-22 8:48 UTC (permalink / raw)
To: Weston Andros Adamson; +Cc: Jeff Layton, linux-nfs list, Simo Sorce
On 21/10/13 22:38, Weston Andros Adamson wrote:
>
> On Oct 21, 2013, at 10:31 PM, Weston Andros Adamson <dros@netapp.com>
> wrote:
>
>>
>> On Oct 21, 2013, at 10:23 PM, Jeff Layton <jlayton@poochiereds.net> wrote:
>>
>>> On Mon, 21 Oct 2013 23:53:16 +0000
>>> Weston Andros Adamson <dros@netapp.com> wrote:
>>>
>>>> I traced this behavior back to:
>>>>
>>>> commit 302de786930a2c533068f9d8909a817b40f07c32
>>>> Author: Simo Sorce <simo@redhat.com>
>>>> Date: Fri Apr 19 13:02:36 2013 -0400
>>>>
>>>> gssd: Allow GSSAPI to try to acquire credentials first.
>>>>
>>>>
>>>> And in particular:
>>>>
>>>> - for (dirname = ccachesearch; *dirname != NULL; dirname++) {
>>>> + /* Try first to acquire credentials directly via GSSAPI */
>>>> + err = gssd_acquire_user_cred(uid, &gss_cred);
>>>> + if (!err)
>>>> + create_resp = create_auth_rpc_client(clp, &rpc_clnt, &auth, uid,
>>>> + AUTHTYPE_KRB5, gss_cred);
>>>> + /* if create_auth_rplc_client fails try the traditional method of
>>>> + * trolling for credentials */
>>>> + for (dirname = ccachesearch; create_resp != 0 && *dirname != NULL; dirname++) {
>>>>
>>>
>>>
>>>> A couple of things:
>>>>
>>>> - If I get rid of the "Try first to acquire credentials directly via GSSAPI" part, expiry works as before.
>>>>
>>>
>>>
>>> Steve just merged a couple of patches from me that change this code
>>> some. It's probably worth testing with those before you make any
>>> changes.
>>>
>>
>> Thanks, I'll check it out.
>
> Bisecting brought me to 302de786930a2c533068f9d8909a817b40f07c32 and I've confirmed that the problem is still in steved's master branch as of today. Are you sure the patches you're thinking of have been merged?
I believe the ones in question are
commit 6b53fc9ce38ba6fff2fd5c2f6ed143747067a39d
Author: Jeff Layton <jlayton@redhat.com>
Date: Mon Oct 21 13:28:06 2013 -0400
gssd: do a more thorough change of identity after forking
commit f9cac65972da588d5218236de60a7be11247a8aa
Author: Jeff Layton <jlayton@redhat.com>
Date: Mon Oct 21 13:27:22 2013 -0400
gssd: have process_krb5_upcall fork before handling upcall
Which have been committed...
steved.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Strange cred expiry behavior
2013-10-21 23:53 ` Weston Andros Adamson
2013-10-22 2:23 ` Jeff Layton
@ 2013-10-22 12:20 ` Simo Sorce
2013-10-22 12:52 ` Weston Andros Adamson
1 sibling, 1 reply; 10+ messages in thread
From: Simo Sorce @ 2013-10-22 12:20 UTC (permalink / raw)
To: Weston Andros Adamson; +Cc: linux-nfs list
On Mon, 2013-10-21 at 23:53 +0000, Weston Andros Adamson wrote:
> I traced this behavior back to:
>
> commit 302de786930a2c533068f9d8909a817b40f07c32
> Author: Simo Sorce <simo@redhat.com>
> Date: Fri Apr 19 13:02:36 2013 -0400
>
> gssd: Allow GSSAPI to try to acquire credentials first.
>
>
> And in particular:
>
> - for (dirname = ccachesearch; *dirname != NULL; dirname++) {
> + /* Try first to acquire credentials directly via GSSAPI */
> + err = gssd_acquire_user_cred(uid, &gss_cred);
> + if (!err)
> + create_resp = create_auth_rpc_client(clp, &rpc_clnt, &auth, uid,
> + AUTHTYPE_KRB5, gss_cred);
> + /* if create_auth_rplc_client fails try the traditional method of
> + * trolling for credentials */
> + for (dirname = ccachesearch; create_resp != 0 && *dirname != NULL; dirname++) {
>
> A couple of things:
>
> - If I get rid of the "Try first to acquire credentials directly via
> GSSAPI" part, expiry works as before.
Interesting, this call is necessary to make rpc.gssd work with gss-proxy
and I had not experienced an issue so far. We can try to add a
gss_inquire_cred() to gssd_acquire_user_cred() to force checking the
credentials for expiration, as that check may be deferred in some cases
in GSSAPI.
> - It says "if create_auth_rplc_client fails try the traditional
> method of trolling for credentials" but I see nothing that checks to
> see if create_resp != 0 before falling back to the old method.
It's the check condition in the for loop, that's why I have not repeated
it before the for loop.
> I added this check and everything seems right (it doesn't fix the
> expiry issue).
Yeah I didn't think it would.
> I should have some time tomorrow to look further at the expiry issue…
If you can add an explicit check with gss_inquire_creds() I think you'll
get the right behavior.
Let me know if you need help.
Simo.
--
Simo Sorce * Red Hat, Inc * New York
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Strange cred expiry behavior
2013-10-22 2:38 ` Weston Andros Adamson
2013-10-22 8:48 ` Steve Dickson
@ 2013-10-22 12:23 ` Simo Sorce
1 sibling, 0 replies; 10+ messages in thread
From: Simo Sorce @ 2013-10-22 12:23 UTC (permalink / raw)
To: Weston Andros Adamson; +Cc: Jeff Layton, linux-nfs list
On Tue, 2013-10-22 at 02:38 +0000, Weston Andros Adamson wrote:
> On Oct 21, 2013, at 10:31 PM, Weston Andros Adamson <dros@netapp.com>
> wrote:
>
> >
> > On Oct 21, 2013, at 10:23 PM, Jeff Layton <jlayton@poochiereds.net> wrote:
> >
> >> On Mon, 21 Oct 2013 23:53:16 +0000
> >> Weston Andros Adamson <dros@netapp.com> wrote:
> >>
> >>> I traced this behavior back to:
> >>>
> >>> commit 302de786930a2c533068f9d8909a817b40f07c32
> >>> Author: Simo Sorce <simo@redhat.com>
> >>> Date: Fri Apr 19 13:02:36 2013 -0400
> >>>
> >>> gssd: Allow GSSAPI to try to acquire credentials first.
> >>>
> >>>
> >>> And in particular:
> >>>
> >>> - for (dirname = ccachesearch; *dirname != NULL; dirname++) {
> >>> + /* Try first to acquire credentials directly via GSSAPI */
> >>> + err = gssd_acquire_user_cred(uid, &gss_cred);
> >>> + if (!err)
> >>> + create_resp = create_auth_rpc_client(clp, &rpc_clnt, &auth, uid,
> >>> + AUTHTYPE_KRB5, gss_cred);
> >>> + /* if create_auth_rplc_client fails try the traditional method of
> >>> + * trolling for credentials */
> >>> + for (dirname = ccachesearch; create_resp != 0 && *dirname != NULL; dirname++) {
> >>>
> >>
> >>
> >>> A couple of things:
> >>>
> >>> - If I get rid of the "Try first to acquire credentials directly via GSSAPI" part, expiry works as before.
> >>>
> >>
> >>
> >> Steve just merged a couple of patches from me that change this code
> >> some. It's probably worth testing with those before you make any
> >> changes.
> >>
> >
> > Thanks, I'll check it out.
>
> Bisecting brought me to 302de786930a2c533068f9d8909a817b40f07c32 and
> I've confirmed that the problem is still in steved's master branch as
> of today. Are you sure the patches you're thinking of have been
> merged?
TBH I do not expect those patches to make any difference in this case.
Simo.
--
Simo Sorce * Red Hat, Inc * New York
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Strange cred expiry behavior
2013-10-22 12:20 ` Simo Sorce
@ 2013-10-22 12:52 ` Weston Andros Adamson
2013-10-22 13:59 ` Weston Andros Adamson
0 siblings, 1 reply; 10+ messages in thread
From: Weston Andros Adamson @ 2013-10-22 12:52 UTC (permalink / raw)
To: Simo Sorce; +Cc: linux-nfs list
Thanks Simo!
I'll try doing the gss_inquire_creds() first and if that fixes things.
-dros
On Oct 22, 2013, at 8:20 AM, Simo Sorce <simo@redhat.com>
wrote:
> On Mon, 2013-10-21 at 23:53 +0000, Weston Andros Adamson wrote:
>> I traced this behavior back to:
>>
>> commit 302de786930a2c533068f9d8909a817b40f07c32
>> Author: Simo Sorce <simo@redhat.com>
>> Date: Fri Apr 19 13:02:36 2013 -0400
>>
>> gssd: Allow GSSAPI to try to acquire credentials first.
>>
>>
>> And in particular:
>>
>> - for (dirname = ccachesearch; *dirname != NULL; dirname++) {
>> + /* Try first to acquire credentials directly via GSSAPI */
>> + err = gssd_acquire_user_cred(uid, &gss_cred);
>> + if (!err)
>> + create_resp = create_auth_rpc_client(clp, &rpc_clnt, &auth, uid,
>> + AUTHTYPE_KRB5, gss_cred);
>> + /* if create_auth_rplc_client fails try the traditional method of
>> + * trolling for credentials */
>> + for (dirname = ccachesearch; create_resp != 0 && *dirname != NULL; dirname++) {
>>
>> A couple of things:
>>
>> - If I get rid of the "Try first to acquire credentials directly via
>> GSSAPI" part, expiry works as before.
>
> Interesting, this call is necessary to make rpc.gssd work with gss-proxy
> and I had not experienced an issue so far. We can try to add a
> gss_inquire_cred() to gssd_acquire_user_cred() to force checking the
> credentials for expiration, as that check may be deferred in some cases
> in GSSAPI.
>
>> - It says "if create_auth_rplc_client fails try the traditional
>> method of trolling for credentials" but I see nothing that checks to
>> see if create_resp != 0 before falling back to the old method.
>
> It's the check condition in the for loop, that's why I have not repeated
> it before the for loop.
>
>> I added this check and everything seems right (it doesn't fix the
>> expiry issue).
>
> Yeah I didn't think it would.
>
>> I should have some time tomorrow to look further at the expiry issue…
>
> If you can add an explicit check with gss_inquire_creds() I think you'll
> get the right behavior.
>
> Let me know if you need help.
> Simo.
>
> --
> Simo Sorce * Red Hat, Inc * New York
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Strange cred expiry behavior
2013-10-22 12:52 ` Weston Andros Adamson
@ 2013-10-22 13:59 ` Weston Andros Adamson
0 siblings, 0 replies; 10+ messages in thread
From: Weston Andros Adamson @ 2013-10-22 13:59 UTC (permalink / raw)
To: Simo Sorce; +Cc: linux-nfs list
This works! Posting a patch now.
Thanks,
-dros
On Oct 22, 2013, at 8:52 AM, Weston Andros Adamson <dros@netapp.com> wrote:
> Thanks Simo!
>
> I'll try doing the gss_inquire_creds() first and if that fixes things.
>
> -dros
>
> On Oct 22, 2013, at 8:20 AM, Simo Sorce <simo@redhat.com>
> wrote:
>
>> On Mon, 2013-10-21 at 23:53 +0000, Weston Andros Adamson wrote:
>>> I traced this behavior back to:
>>>
>>> commit 302de786930a2c533068f9d8909a817b40f07c32
>>> Author: Simo Sorce <simo@redhat.com>
>>> Date: Fri Apr 19 13:02:36 2013 -0400
>>>
>>> gssd: Allow GSSAPI to try to acquire credentials first.
>>>
>>>
>>> And in particular:
>>>
>>> - for (dirname = ccachesearch; *dirname != NULL; dirname++) {
>>> + /* Try first to acquire credentials directly via GSSAPI */
>>> + err = gssd_acquire_user_cred(uid, &gss_cred);
>>> + if (!err)
>>> + create_resp = create_auth_rpc_client(clp, &rpc_clnt, &auth, uid,
>>> + AUTHTYPE_KRB5, gss_cred);
>>> + /* if create_auth_rplc_client fails try the traditional method of
>>> + * trolling for credentials */
>>> + for (dirname = ccachesearch; create_resp != 0 && *dirname != NULL; dirname++) {
>>>
>>> A couple of things:
>>>
>>> - If I get rid of the "Try first to acquire credentials directly via
>>> GSSAPI" part, expiry works as before.
>>
>> Interesting, this call is necessary to make rpc.gssd work with gss-proxy
>> and I had not experienced an issue so far. We can try to add a
>> gss_inquire_cred() to gssd_acquire_user_cred() to force checking the
>> credentials for expiration, as that check may be deferred in some cases
>> in GSSAPI.
>>
>>> - It says "if create_auth_rplc_client fails try the traditional
>>> method of trolling for credentials" but I see nothing that checks to
>>> see if create_resp != 0 before falling back to the old method.
>>
>> It's the check condition in the for loop, that's why I have not repeated
>> it before the for loop.
>>
>>> I added this check and everything seems right (it doesn't fix the
>>> expiry issue).
>>
>> Yeah I didn't think it would.
>>
>>> I should have some time tomorrow to look further at the expiry issue…
>>
>> If you can add an explicit check with gss_inquire_creds() I think you'll
>> get the right behavior.
>>
>> Let me know if you need help.
>> Simo.
>>
>> --
>> Simo Sorce * Red Hat, Inc * New York
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2013-10-22 13:59 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-10-21 20:07 Strange cred expiry behavior Weston Andros Adamson
2013-10-21 23:53 ` Weston Andros Adamson
2013-10-22 2:23 ` Jeff Layton
2013-10-22 2:31 ` Weston Andros Adamson
2013-10-22 2:38 ` Weston Andros Adamson
2013-10-22 8:48 ` Steve Dickson
2013-10-22 12:23 ` Simo Sorce
2013-10-22 12:20 ` Simo Sorce
2013-10-22 12:52 ` Weston Andros Adamson
2013-10-22 13:59 ` Weston Andros Adamson
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.