The purpose of this blog post is to describe one such error that I am currently investigating, I will say upfront however I have got to the bottom of why the error is being reported and have a possible workaround for some situations but the quest for a solution is ongoing.
If and when a complete solution is available I will update this blog post with the additional information.
In this case I have a JBoss EAP server handling incoming SPNEGO messages from different clients.
The Errors
Before describing the findings here are a couple of the errors I have been seeing: -Caused by: KrbException: EncryptedData is encrypted using keytype NULL but decryption key is of type AES256 CTS mode with HMAC SHA1-96
at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:169) [rt.jar:1.7.0_17]
at sun.security.krb5.KrbCred.<init>(KrbCred.java:131) [rt.jar:1.7.0_17]
at sun.security.jgss.krb5.InitialToken$OverloadedChecksum.<init>(InitialToken.java:291) [rt.jar:1.7.0_17]
at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:130) [rt.jar:1.7.0_17]
at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:771) [rt.jar:1.7.0_17]
And
Caused by: KrbException: EncryptedData is encrypted using keytype DES3 CBC mode with SHA1-KD but decryption key is of type NULL
at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:169) [rt.jar:1.7.0_17]
at sun.security.krb5.KrbCred.<init>(KrbCred.java:131) [rt.jar:1.7.0_17]
at sun.security.jgss.krb5.InitialToken$OverloadedChecksum.<init>(InitialToken.java:282) [rt.jar:1.7.0_17]
at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:130) [rt.jar:1.7.0_17]
at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:771) [rt.jar:1.7.0_17]
... 35 more
For the purpose of identifying if your error message is related to this it is absolutely essential that you also compare the stack trace to ensure it is similar, i.e. the line numbers may be different but the classes in the stack should be almost identical.
The first of these error messages was reported when connecting to a server using Firefox from a Mac OS client, the second was reported using Firefox on a Fedora 20 client.
The Trigger
Within the SPNEGO authentication a Kerberos AP_REQ message is sent from the client to the server, within this message it is possible for the client to embed their own credential so that it can be delegated to the server. For both of the error message above the KrbCred creation, EncryptedData decryption and the resulting validation and error are only applicable if the incoming message does contain a delegated credential.
When it comes to credential delegation and Kerberos this is something that the client decides it is going to do, the server has not actually requested that the client does delegate their credentials.
The Workaround
As this error is only encountered where credentials are delegated to the server the most simple workaround is to disable credential delegation in your client. That way the AP_REQ message will not contain delegated credentials and the portion of the stack above reporting the error message will not be hit.
Unfortunately this workaround is only suitable if your server side service is not making use of delegated credentials but many services use SPNEGO without requiring delgation.
The Trigger II
So what is actually happening during the parsing that is triggering the error.
Within the Kerberos protocol a session key is established, certain fields can be encrypted using the session key and on parsing the message this session key can be used to decrypt the field.
The encoded representation of an encrypted attribute contains a field that specifies the encryption type in use, for the above errors we see three types mentioned aes256-cts-hmac-sha1-96, des3-cbc-sha1, and also NULL. Null is a special case as it specified that an encrypted field is not actually encrypted.
So back to the Java implementation, before parsing and possibly decrypting the delegated credential the Java implementation checks the type of the agreed session key. If the session key is for aes256-cts-hmac-sha1-96, aes128-cts-hmac-sha1-96, or is arcfour then an assumption is made that the session key should be used to decrypt the credential. For all other cases it assumes the encryption type will be NULL and that a NULL key should be used for the decryption.
So for the first error I posted above the Mac OS client used NULL for encrypting the delegated credential however as the session key is aes256-cts-hmac-sha1-96 the parsing is expecting this to have been used.
For the second error with the Linux client the session key is now des3-cbc-sha1, the Linux client did encrypt the delegated credential but the Java implementation has made the assumption it should not be doing that.
The Next Steps
As I said at the start at the time of posting this I do not have the solution, only the information that will hopefully make it easier for you to see what the error means and one possible workaround.
Overall this error is as a result of different approaches to message creation and parsing between the different clients and the Java implementation - so now I need to get to the bottom of which side is actually at fault. My initial suspicion is the Java side as it should be possibly to identify the encryption type that was used before this error is reported but that is going to require some further investigation and discussions.
