[BUG] sec=krb5 mount problem with nfs-utils 1.2.3 on client side

[BUG] sec=krb5 mount problem with nfs-utils 1.2.3 on client side

[Michael Guntsche]

Hi,

I recently updated my nfs clients and server to the new nfs-utls Version
1.2.3 and then also tested a sec=krb5 mount via nfs4 again. For some reason
the mount failed and I got the following message on the server.

rpc.svcgssd output:
===================
entering poll
leaving poll
handling null request
sname = nfs/zaphod.comsick.at@COMSICK.AT
DEBUG: serialize_krb5_ctx: lucid version!
prepare_krb5_rfc4121_buffer: protocol 1
prepare_krb5_rfc4121_buffer: serializing key with enctype 18 and size 32
doing downcall
mech: krb5, hndl len: 4, ctx len 52, timeout: 1302858643 (35979 from
now), clnt: nfs@zaphod.comsick.at, uid: -1, gid: -1, num aux grps: 0:
: qword_eol: fflush failed: errno 22 (Invalid argument)
WARNING: error writing to downcall channel
/proc/net/rpc/auth.rpcsec.context/channel: Invalid argument
sending null reply
finished handling null request
<repeated 4 times>

Since it worked before I downgraded the versions on both the client and
the server to see when it started working again.

Server with 1.2.3 and  client with 1.2.2 did the trick

rpc.svcgssd output:
===================
entering poll
leaving poll
handling null request
sname = nfs/zaphod.comsick.at@COMSICK.AT
DEBUG: serialize_krb5_ctx: lucid version!
prepare_krb5_rfc1964_buffer: serializing keys with enctype 4 and length
8
doing downcall
mech: krb5, hndl len: 4, ctx len 85, timeout: 1302858893 (36000 from
now), clnt: nfs@zaphod.comsick.at, uid: -1, gid: -1, num aux grps: 0:
sending null reply
finished handling null request
entering poll

So apparently with 1.2.2 on the client side a different enctype is
select. I searched in the archives and saw that this problem was already
known but I did not see any fixes. Both client and server have
aes_generic enabled and loaded I also made sure that the correct keys on
the kerberos side are available.

Valid starting     Expires            Service principal 04/15/11
01:14:53  04/15/11 11:14:53  krbtgt/COMSICK.AT@COMSICK.AT renew until
04/16/11 01:14:53, Etype (skey, tkt): des3-cbc-sha1, des3-cbc-sha1
04/15/11 01:14:53  04/15/11 11:14:53  nfs/gibson.comsick.at@COMSICK.AT
renew until 04/16/11 01:14:53, Etype (skey, tkt): des-cbc-crc,
aes256-cts-hmac-sha1-96 

So right now I am little bit stumped 

* why another enctype is used
* why it does not succeed with the new enctype

Both machines are running kernel 2.6.38 on debian sid with the latest
Kerberos 1.9

If you need more information please tell me.

Kind regards,
Michael Guntsche

Re: [BUG] sec=krb5 mount problem with nfs-utils 1.2.3 on client side

[Vladimir Elisseev]

Michael,

I've posted an email with the same question here a few weeks ago, but I
haven't been able to investigate what the problem is. The only
difference in my setup, that servers have kernel 2.6.36 the rest is
exactly the same and exactly the same error. See thread with the subject
"rpc.svcgssd problem after updating client 1.2.2->1.2.3".

Regards,
Vladimir.

On Fri, 2011-04-15 at 01:20 +0200, Michael Guntsche wrote:
> Hi,
> 
> I recently updated my nfs clients and server to the new nfs-utls Version
> 1.2.3 and then also tested a sec=krb5 mount via nfs4 again. For some reason
> the mount failed and I got the following message on the server.
> 
> rpc.svcgssd output:
> ===================
> entering poll
> leaving poll
> handling null request
> sname = nfs/zaphod.comsick.at@COMSICK.AT
> DEBUG: serialize_krb5_ctx: lucid version!
> prepare_krb5_rfc4121_buffer: protocol 1
> prepare_krb5_rfc4121_buffer: serializing key with enctype 18 and size 32
> doing downcall
> mech: krb5, hndl len: 4, ctx len 52, timeout: 1302858643 (35979 from
> now), clnt: nfs@zaphod.comsick.at, uid: -1, gid: -1, num aux grps: 0:
> : qword_eol: fflush failed: errno 22 (Invalid argument)
> WARNING: error writing to downcall channel
> /proc/net/rpc/auth.rpcsec.context/channel: Invalid argument
> sending null reply
> finished handling null request
> <repeated 4 times>
> 
> Since it worked before I downgraded the versions on both the client and
> the server to see when it started working again.
> 
> Server with 1.2.3 and  client with 1.2.2 did the trick
> 
> rpc.svcgssd output:
> ===================
> entering poll
> leaving poll
> handling null request
> sname = nfs/zaphod.comsick.at@COMSICK.AT
> DEBUG: serialize_krb5_ctx: lucid version!
> prepare_krb5_rfc1964_buffer: serializing keys with enctype 4 and length
> 8
> doing downcall
> mech: krb5, hndl len: 4, ctx len 85, timeout: 1302858893 (36000 from
> now), clnt: nfs@zaphod.comsick.at, uid: -1, gid: -1, num aux grps: 0:
> sending null reply
> finished handling null request
> entering poll
> 
> So apparently with 1.2.2 on the client side a different enctype is
> select. I searched in the archives and saw that this problem was already
> known but I did not see any fixes. Both client and server have
> aes_generic enabled and loaded I also made sure that the correct keys on
> the kerberos side are available.
> 
> Valid starting     Expires            Service principal 04/15/11
> 01:14:53  04/15/11 11:14:53  krbtgt/COMSICK.AT@COMSICK.AT renew until
> 04/16/11 01:14:53, Etype (skey, tkt): des3-cbc-sha1, des3-cbc-sha1
> 04/15/11 01:14:53  04/15/11 11:14:53  nfs/gibson.comsick.at@COMSICK.AT
> renew until 04/16/11 01:14:53, Etype (skey, tkt): des-cbc-crc,
> aes256-cts-hmac-sha1-96 
> 
> So right now I am little bit stumped 
> 
> * why another enctype is used
> * why it does not succeed with the new enctype
> 
> Both machines are running kernel 2.6.38 on debian sid with the latest
> Kerberos 1.9
> 
> If you need more information please tell me.
> 
> Kind regards,
> Michael Guntsche
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] sec=krb5 mount problem with nfs-utils 1.2.3 on client side

[Finney, Sean]

SGkgZ3V5cywKCk9uIEZyaSwgMjAxMS0wNC0xNSBhdCAwNjowMiArMDIwMCwgVmxhZGltaXIgRWxp
c3NlZXYgd3JvdGU6Cj4gTWljaGFlbCwKPiAKPiBJJ3ZlIHBvc3RlZCBhbiBlbWFpbCB3aXRoIHRo
ZSBzYW1lIHF1ZXN0aW9uIGhlcmUgYSBmZXcgd2Vla3MgYWdvLCBidXQgSQo+IGhhdmVuJ3QgYmVl
biBhYmxlIHRvIGludmVzdGlnYXRlIHdoYXQgdGhlIHByb2JsZW0gaXMuIFRoZSBvbmx5Cj4gZGlm
ZmVyZW5jZSBpbiBteSBzZXR1cCwgdGhhdCBzZXJ2ZXJzIGhhdmUga2VybmVsIDIuNi4zNiB0aGUg
cmVzdCBpcwo+IGV4YWN0bHkgdGhlIHNhbWUgYW5kIGV4YWN0bHkgdGhlIHNhbWUgZXJyb3IuIFNl
ZSB0aHJlYWQgd2l0aCB0aGUgc3ViamVjdAo+ICJycGMuc3ZjZ3NzZCBwcm9ibGVtIGFmdGVyIHVw
ZGF0aW5nIGNsaWVudCAxLjIuMi0+MS4yLjMiLgoKSSB0aGluayB0aGUgcHJvYmxlbSBoZXJlIGlz
IHRoZSBzYW1lIG9uZSB3ZSB3ZXJlIGV4cGVyaWVuY2luZywgd2l0aApsYXJnZSBncm91cCBtZW1i
ZXJzaGlwcyBjYXVzaW5nIGludmFsaWQgd3JpdGVzIHRvIHRoZSBwcm9jZnMgcnBjIGNoYW5uZWwK
ZmlsZXMuICBJZiB5b3Ugc3RyYWNlIHJwYy5zdmNnc3NkIChvciBycGMubW91bnRkLCB5b3Ugc2hv
dWxkIHNlZSBpdAp0aGVyZSB0b28gSSB0aGluayksIHlvdSBzaG91bGQgc2VlIHRoZSB3cml0ZSgp
IGNhbGxzIGJlaW5nIHNwbGl0IGludG8KbXVsdGlwbGUgMTAyNC1ieXRlIHNpemVkIHdyaXRlcywg
ZWFjaCBvZiB3aGljaCBnZXQgRUlOVkFMIChvciAtRUlOVkFMLCBJCmZvcmdldCkgcmV0dXJuZWQu
CgpJJ3ZlIHN1Ym1pdHRlZCBhIHBhdGNoIGZvciB0aGF0IGxhc3Qgd2Vlaywgd2hpY2ggaXMgd2Fp
dGluZyBvbgppbmNsdXNpb24uICBJZiB5b3Ugd2FudCB0byB2ZXJpZnksIHBsZWFzZSBsb29rIGJh
Y2sgaW4gdGhlIGFyY2hpdmUgYW5kCnRyeSB0aGUgcGF0Y2ggc3VibWl0dGVkIGluIHRoZSBtYWls
IHdpdGggc3ViamVjdDoKCltQQVRDSCB2Ml0gbmZzLXV0aWxzOiBJbmNyZWFzZSB0aGUgc3RkaW8g
ZmlsZSBidWZmZXIgc2l6ZSBmb3IgcHJvY2ZzCmZpbGVzCgpJdCBzZWVtcyB0byB3b3JrIGhlcmUs
IEknZCBhcHByZWNpYXRlIGlmIHlvdSBjb3VsZCB0cnkgaXQgb3V0IGFuZCBsZXQgbWUKa25vdyBp
ZiBpdCB3b3JrcyBmb3IgeW91IHRvbyA6KQoKCkJSCglTZWFuCgTvv717Lm7vv70r77+977+977+9
77+977+977+977+9KyXvv73vv71sendt77+977+9Yu+/veunsu+/ve+/vXLvv73vv716WO+/ve+/
vRnfsinvv73vv73vv713Kh9qZ++/ve+/ve+/vR7vv73vv73vv73vv73vv73domov77+977+977+9
eu+/vd6W77+977+9Mu+/vd6Z77+977+977+9Ju+/vSnfoe+/vWHvv73vv71/77+977+9Hu+/vUfv
v73vv73vv71o77+9D++/vWo6K3bvv73vv73vv71377+92aU=

Re: [BUG] sec=krb5 mount problem with nfs-utils 1.2.3 on client side

[Vladimir Elisseev]

Sean,

I'm pretty sure this is not the case for my environment, as we have  
less then 100 groups in total.

Regards,
Vladimir.

Quoting "Finney, Sean" <Sean.Finney@sonyericsson.com>:

> Hi guys,
>
> On Fri, 2011-04-15 at 06:02 +0200, Vladimir Elisseev wrote:
>> Michael,
>>
>> I've posted an email with the same question here a few weeks ago, but I
>> haven't been able to investigate what the problem is. The only
>> difference in my setup, that servers have kernel 2.6.36 the rest is
>> exactly the same and exactly the same error. See thread with the subject
>> "rpc.svcgssd problem after updating client 1.2.2->1.2.3".
>
> I think the problem here is the same one we were experiencing, with
> large group memberships causing invalid writes to the procfs rpc channel
> files.  If you strace rpc.svcgssd (or rpc.mountd, you should see it
> there too I think), you should see the write() calls being split into
> multiple 1024-byte sized writes, each of which get EINVAL (or -EINVAL, I
> forget) returned.
>
> I've submitted a patch for that last week, which is waiting on
> inclusion.  If you want to verify, please look back in the archive and
> try the patch submitted in the mail with subject:
>
> [PATCH v2] nfs-utils: Increase the stdio file buffer size for procfs
> files
>
> It seems to work here, I'd appreciate if you could try it out and let me
> know if it works for you too :)
>
>
> BR
> 	Sean
>

Re: [BUG] sec=krb5 mount problem with nfs-utils 1.2.3 on client side

[Finney, Sean]

SGkgVmxhZGltaXIsCgpPbiBGcmksIDIwMTEtMDQtMTUgYXQgMDk6MTUgKzAyMDAsIFZsYWRpbWly
IEVsaXNzZWV2IHdyb3RlOgo+IEknbSBwcmV0dHkgc3VyZSB0aGlzIGlzIG5vdCB0aGUgY2FzZSBm
b3IgbXkgZW52aXJvbm1lbnQsIGFzIHdlIGhhdmUgIAo+IGxlc3MgdGhlbiAxMDAgZ3JvdXBzIGlu
IHRvdGFsLgoKUmUtcmVhZGluZyB0aGUgbWFpbCwgSSByZWFsaXplIHlvdSBtaWdodCBiZSByaWdo
dC4gIFRoZSBFSU5WQUxJRCBvbiB0aGUKcnBjIGNoYW5uZWwgZmlsZSBqdW1wZWQgb3V0IGF0IG1l
IHRob3VnaCBzaW5jZSBpdCB3YXMgdGhlIHNhbWUgc3ltcHRvbQppbiBvdXIgY2FzZS4gIEFsc28s
IGluIG91ciBjYXNlIHdlIHNhdyB0aGUgcHJvYmxlbSBpbiBhbGwgdmVyc2lvbnMgb2YKbmZzLXV0
aWxzIGFuZCBub3QganVzdCB0aGUgbGF0ZXN0IG9uZSwgc28gaXQgY291bGQgYmUgdGhhdCBpdCdz
IGFuIGVycm9yCnNpbWlsYXIgaW4gbmF0dXJlIGJ1dCBub3QgdGhlIHNhbWUgdW5kZXJseWluZyBj
YXVzZS4KCkFueXdheSwgaW4gY2FzZSBpdCdzIGFueSBoZWxwIChhc3N1bWluZyB0aGF0IHRoZSBw
YXRjaCBkb2Vzbid0IGZpeCB0aGUKcHJvYmxlbSBmb3IgeW91KSwgdGhlIGtlcm5lbCByZXR1cm5z
IEVJTlZBTElEIG9uIHdyaXRlKCkncyB0byB0aGUgZmlsZQp3aGVuIGFueSBzaW5nbGUgd3JpdGUg
aXMgbm90IHByb3Blcmx5IGZvcm1hdHRlZCBhbmQgbmV3bGluZSB0ZXJtaW5hdGVkLgpJdCBzZWVt
cyB0byBiZSB2ZXJ5IHBpY2t5IGFib3V0IHRoYXQuICBJZiB5b3Ugc3RyYWNlIC1zMjA0OCB0aGUK
b2ZmZW5kaW5nIGRhZW1vbiwgaXQgc2hvdWxkIGJlIHByZXR0eSBjbGVhciB0aG91Z2gsIGVpdGhl
ciB5b3UnbGwgc2VlIGEKc2VyaWVzIG9mIHNwbGl0IHVwIEJVRlNaIHNpemVkIHdyaXRlcyAoZWFj
aCBnZXR0aW5nIEVJTlZBTElEKSBvciB5b3UnbGwKc2VlIGEgd3JpdGUgb3BlcmF0aW9uIHRoYXQn
cyBvdGhlcndpc2Ugc29tZWhvdyBpbXByb3Blcmx5IGZvcm1hdHRlZC4KCkhvcGUgdGhhdCdzIGF0
IGxlYXN0IGEgYml0IGhlbHBmdWwsCgoJU2VhbgoT77+977+97Lm7HO+/vSbvv71+77+9Ju+/vRjv
v73vv70rLe+/ve+/vd22F++/ve+/vXfvv73vv73Lm++/ve+/ve+/vW3vv71i77+977+9Z37Ip++/
vRfvv73vv73cqH3vv73vv73vv73GoHrvv70majordu+/ve+/ve+/vQfvv73vv73vv73vv716Wivv
v73vv70rembvv73vv73vv71o77+977+977+9fu+/ve+/ve+/ve+/vWnvv73vv73vv71677+9Hu+/
vXfvv73vv73vv70/77+977+977+977+9Ju+/vSnfohtm

Re: [BUG] sec=krb5 mount problem with nfs-utils 1.2.3 on client side

[Michael Guntsche]

On 15 Apr 11 08:01, Finney, Sean wrote:
> Hi guys,
> 
> On Fri, 2011-04-15 at 06:02 +0200, Vladimir Elisseev wrote:
> > Michael,
> > 
> > I've posted an email with the same question here a few weeks ago, but I
> > haven't been able to investigate what the problem is. The only
> > difference in my setup, that servers have kernel 2.6.36 the rest is
> > exactly the same and exactly the same error. See thread with the subject
> > "rpc.svcgssd problem after updating client 1.2.2->1.2.3".
> 
> I think the problem here is the same one we were experiencing, with
> large group memberships causing invalid writes to the procfs rpc channel
> files.  If you strace rpc.svcgssd (or rpc.mountd, you should see it
> there too I think), you should see the write() calls being split into
> multiple 1024-byte sized writes, each of which get EINVAL (or -EINVAL, I
> forget) returned.
<snip>

Thank you for the information, but I got it working in the meantime.
The main problem still is that the code for some reason tries to use AES
although I tried specifying a different enctype in my kerberos config.
Nevertheless it should just work with AES as well, so where was the
problem? 
Quite simple....missing kernel support. I enabled AES support but I DID
NOT enable CTS support which is of course needed as well. So after
compiling the server and client kernels with BOTH AES and CTS support I
can no mount the NFS4 export without any issues.

I still think the error message could be a little bit more
verbose/meaningful like "Missing kernel support for X" instead of a
rather cryptic message.

Once again, sorry for the noise and I'll keep testing this setup now
more often so this hopefully does not happen again.

Kind regards,
Michael Guntsche

Re: [BUG] sec=krb5 mount problem with nfs-utils 1.2.3 on client side

[Trond Myklebust]

On Fri, 2011-04-15 at 12:16 +0200, Michael Guntsche wrote:
> Thank you for the information, but I got it working in the meantime.
> The main problem still is that the code for some reason tries to use AES
> although I tried specifying a different enctype in my kerberos config.
> Nevertheless it should just work with AES as well, so where was the
> problem? 
> Quite simple....missing kernel support. I enabled AES support but I DID
> NOT enable CTS support which is of course needed as well. So after
> compiling the server and client kernels with BOTH AES and CTS support I
> can no mount the NFS4 export without any issues.

Sigh. We really should not allow that kind of config. It just creates
confusion.

Kevin, what are the dependencies for the kerberos V module today? Am I
missing something in the following list?

        depends on SUNRPC && CRYPTO
        depends on CRYPTO_MD5 && CRYPTO_DES && CRYPTO_CBC && CRYPTO_CTS
        depends on CRYPTO_ECB && CRYPTO_HMAC && CRYPTO_MD5 &&
        CRYPTO_SHA1
        depends on CRYPTO_AES

Cheers
 Trond

Re: [BUG] sec=krb5 mount problem with nfs-utils 1.2.3 on client side

[Trond Myklebust]

On Fri, 2011-04-15 at 09:29 -0400, Trond Myklebust wrote:
> On Fri, 2011-04-15 at 12:16 +0200, Michael Guntsche wrote:
> > Thank you for the information, but I got it working in the meantime.
> > The main problem still is that the code for some reason tries to use AES
> > although I tried specifying a different enctype in my kerberos config.
> > Nevertheless it should just work with AES as well, so where was the
> > problem? 
> > Quite simple....missing kernel support. I enabled AES support but I DID
> > NOT enable CTS support which is of course needed as well. So after
> > compiling the server and client kernels with BOTH AES and CTS support I
> > can no mount the NFS4 export without any issues.
> 
> Sigh. We really should not allow that kind of config. It just creates
> confusion.
> 
> Kevin, what are the dependencies for the kerberos V module today? Am I
> missing something in the following list?
> 
>         depends on SUNRPC && CRYPTO
>         depends on CRYPTO_MD5 && CRYPTO_DES && CRYPTO_CBC && CRYPTO_CTS
>         depends on CRYPTO_ECB && CRYPTO_HMAC && CRYPTO_MD5 &&

I suppose we only need one test for CRYPTO_MD5...

>         CRYPTO_SHA1
>         depends on CRYPTO_AES
> 
> Cheers
>  Trond
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] sec=krb5 mount problem with nfs-utils 1.2.3 on client side

[Kevin Coffman]

On Fri, Apr 15, 2011 at 9:29 AM, Trond Myklebust
<trond.myklebust@fys.uio.no> wrote:
> On Fri, 2011-04-15 at 12:16 +0200, Michael Guntsche wrote:
>> Thank you for the information, but I got it working in the meantime.
>> The main problem still is that the code for some reason tries to use AES
>> although I tried specifying a different enctype in my kerberos config.
>> Nevertheless it should just work with AES as well, so where was the
>> problem?
>> Quite simple....missing kernel support. I enabled AES support but I DID
>> NOT enable CTS support which is of course needed as well. So after
>> compiling the server and client kernels with BOTH AES and CTS support I
>> can no mount the NFS4 export without any issues.
>
> Sigh. We really should not allow that kind of config. It just creates
> confusion.
>
> Kevin, what are the dependencies for the kerberos V module today? Am I
> missing something in the following list?
>
>        depends on SUNRPC && CRYPTO
>        depends on CRYPTO_MD5 && CRYPTO_DES && CRYPTO_CBC && CRYPTO_CTS
>        depends on CRYPTO_ECB && CRYPTO_HMAC && CRYPTO_MD5 &&
>        CRYPTO_SHA1
>        depends on CRYPTO_AES
>
> Cheers
>  Trond

Yeah, I think that stuff got left out of the final patches.

DES3 needs (in addition to the stuff already there for DES) HMAC and SHA1
AES needs SHA1 AES CTS
RC4 needs ECB ARC4 MD5

So I think you are only missing CRYPTO_ARC4.