Kerberos EncryptedData NULL key / keytype mismatch.

For anyone who has ever been in the position of debugging Kerberos interoperability issues where Java is concerned I am fairly sure you would have come across the situation where you are attempting to search for some meaning in a cryptic error message you have received just to stumble on message posts from other users with the same error but no explanation as to the cause or a solution.

The purpose of this blog post is to describe one such error that I am currently investigating, I will say upfront however I have got to the bottom of why the error is being reported and have a possible workaround for some situations but the quest for a solution is ongoing.

If and when a complete solution is available I will update this blog post with the additional information.

In this case I have a JBoss EAP server handling incoming SPNEGO messages from different clients.

The Errors

Before describing the findings here are a couple of the errors I have been seeing: -

Caused by: KrbException: EncryptedData is encrypted using keytype NULL but decryption key is of type AES256 CTS mode with HMAC SHA1-96
at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:169) [rt.jar:1.7.0_17]
at sun.security.krb5.KrbCred.<init>(KrbCred.java:131) [rt.jar:1.7.0_17]
at sun.security.jgss.krb5.InitialToken$OverloadedChecksum.<init>(InitialToken.java:291) [rt.jar:1.7.0_17]
at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:130) [rt.jar:1.7.0_17]
at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:771) [rt.jar:1.7.0_17]

And

Caused by: KrbException: EncryptedData is encrypted using keytype DES3 CBC mode with SHA1-KD but decryption key is of type NULL
at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:169) [rt.jar:1.7.0_17]
at sun.security.krb5.KrbCred.<init>(KrbCred.java:131) [rt.jar:1.7.0_17]
at sun.security.jgss.krb5.InitialToken$OverloadedChecksum.<init>(InitialToken.java:282) [rt.jar:1.7.0_17]
at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:130) [rt.jar:1.7.0_17]
at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:771) [rt.jar:1.7.0_17]
... 35 more

For the purpose of identifying if your error message is related to this it is absolutely essential that you also compare the stack trace to ensure it is similar, i.e. the line numbers may be different but the classes in the stack should be almost identical.

The first of these error messages was reported when connecting to a server using Firefox from a Mac OS client, the second was reported using Firefox on a Fedora 20 client.

The Trigger

Within the SPNEGO authentication a Kerberos AP_REQ message is sent from the client to the server, within this message it is possible for the client to embed their own credential so that it can be delegated to the server.  For both of the error message above the KrbCred creation, EncryptedData decryption and the resulting validation and error are only applicable if the incoming message does contain a delegated credential.

When it comes to credential delegation and Kerberos this is something that the client decides it is going to do, the server has not actually requested that the client does delegate their credentials.

The Workaround

As this error is only encountered where credentials are delegated to the server the most simple workaround is to disable credential delegation in your client.  That way the AP_REQ message will not contain delegated credentials and the portion of the stack above reporting the error message will not be hit.

Unfortunately this workaround is only suitable if your server side service is not making use of delegated credentials but many services use SPNEGO without requiring delgation.

The Trigger II

So what is actually happening during the parsing that is triggering the error.

Within the Kerberos protocol a session key is established, certain fields can be encrypted using the session key and on parsing the message this session key can be used to decrypt the field.  

The encoded representation of an encrypted attribute contains a field that specifies the encryption type in use, for the above errors we see three types mentioned aes256-cts-hmac-sha1-96, des3-cbc-sha1, and also NULL.  Null is a special case as it specified that an encrypted field is not actually encrypted.

So back to the Java implementation, before parsing and possibly decrypting the delegated credential the Java implementation checks the type of the agreed session key.  If the session key is for aes256-cts-hmac-sha1-96, aes128-cts-hmac-sha1-96, or is arcfour then an assumption is made that the session key should be used to decrypt the credential.  For all other cases it assumes the encryption type will be NULL and that a NULL key should be used for the decryption.

So for the first error I posted above the Mac OS client used NULL for encrypting the delegated credential however as the session key is aes256-cts-hmac-sha1-96 the parsing is expecting this to have been used.

For the second error with the Linux client the session key is now des3-cbc-sha1, the Linux client did encrypt the delegated credential but the Java implementation has made the assumption it should not be doing that.

The Next Steps

As I said at the start at the time of posting this I do not have the solution, only the information that will hopefully make it easier for you to see what the error means and one possible workaround.

Overall this error is as a result of different approaches to message creation and parsing between the different clients and the Java implementation - so now I need to get to the bottom of which side is actually at fault.  My initial suspicion is the Java side as it should be possibly to identify the encryption type that was used before this error is reported but that is going to require some further investigation and discussions.

WildFly 8 Security Realms - LDAP Caching

Another feature added to WildFly 8 that you may not be aware of unless you have gone looking for it is the ability to define caches on security realms where LDAP is being used for authentication or group loading.  This was added under WFLY-1523.

If you are familiar with the caching already in place for JAAS based security domains please be aware that the approach taken for the security realms is very different, for JAAS the key to the cache is essentially using the users supplied username and password and the cached value is the entire interaction with LDAP including all of the groups recursively found.  Within the security realms the caching is much more fine grained and it is the results for individual queries which are cached independently rather than the overall result cached as one.  This will be explained further in this post, for now this essentially means that it is possible that the results from queries for one user can be re-used for another user - this would occur where iteratively querying the group membership information of groups.

Base Configuration

Before describing how to enable caching it makes sense to start with an existing security realm already configured to authenticate users against LDAP and load their group membership information from LDAP, the following is one such possible configuration: -

    "core-service" : {

        "management" : {

            "security-realm" : {

                "LDAPRealm" : {

                    "authentication" : {"ldap" : {

                        "allow-empty-passwords" : false,

                        "base-dn" : "...",

                        "connection" : "MyLdapConnection",

                        "recursive" : false,

                        "user-dn" : "dn",

                        "username-attribute" : "uid",

                        "cache" : null

                    }},

                    "authorization" : {"ldap" : {

                        "connection" : "MyLdapConnection",

                        "group-search" : {"group-to-principal" : {

                            "base-dn" : "...",

                            "group-dn-attribute" : "dn",

                            "group-name" : "SIMPLE",

                            "group-name-attribute" : "uid",

                            "iterative" : true,

                          "principal-attribute" : "uniqueMember",

                            "search-by" : "DISTINGUISHED_NAME",

                            "cache" : null

                        }},

                        "username-to-dn" : {"username-filter" : {

                            "attribute" : "uid",

                            "base-dn" : "...",

                            "force" : false,

                            "recursive" : false,

                            "user-dn-attribute" : "dn",

                            "cache" : null

                        }}

                    }},

                }

            }

        }

This example has been cleaned up slightly but the main point to note is that this configuration has three key areas.

• Authentication

If you are already using LDAP with security realms this should be the most familiar, effectively during authentication we discover the users distinguished name using this definition and attempt to connect to LDAP using their supplied credential to verify they are who they claim to be.

• A username-to-dn definition in group search

When it comes to group searching we may rely on the availability of the users distinguished name, this block is not used in all situations but essentially it is a second attempt to discover a users distinguished name, this is more likely to be required if a second form of authentication was supported e.g. local authentication.

• A group-to-principal group search

Then finally there is the group search definition, in this case it is an iterative search - what that means is that first all of the groups will be identified that the user is directly a member of - after that a search will be performed for each of those groups to identify the groups that those groups are a member of, this process continues until either a cyclic reference is detected or the final groups are not members of any further groups.

Before moving onto the next section please note in the above example there are three points that have a sub resource of "cache" : null , it is at each of these points that a cache can be defined to cache the LDAP interaction for that portion of the model.

The Caches

When enabling a cache it is possible to chose from one of two eviction strategies, the first being by access time and the second by search time.  If you choose by access time then the item in the cache will be evicted from the cache after a certain time period has elapsed since it was last used, if you choose by search time it will be evicted after the configured time has elapsed since the item was added to the cache regardless of if it has been accessed later.

In addition to choosing the eviction strategy the following attributes can also be set on a cache: -

• eviction-time

This one should be fairly self explanatory and is the time in seconds used for evictions depending on the chosen strategy. 

• cache-failures

This one is more important for the cache used in the authentication section and controls if the results of failed LDAP searches are cached to prevent the LDAP server from being hit again - this risk here is that without the next attribute a remote user could fill up the cache by trying many different users that do not exist.

• max-cache-size

In addition to the time based strategy it is also possible to define a maximum size for the cache, if the maximum size of the cache is reached the oldest item in the cache will be evicted to make room for a new item being added - this is the final level of protection to prevent a cache that is caching failures from using up all of the available heap space to cache those failures.

Enabling The Cache 

So that is the description of the different configuration choices you can make, the following is the CLI command you can use to add a cache for the authentication related LDAP access: -

./core-service=management/security-realm=LDAPRealm/authentication=ldap/cache=by-access-time:add(eviction-time=300, cache-failures=true, max-cache-size=100)

This command enabled a cache that caches by access time with items automatically evicted after 5 minutes unless the maximum capacity of 100 items is reached - in addition to that failures are cached so the LDAP server will not be hit repeatedly for authentication attempts where a user is using an invalid user ID.

If the eviction by search time strategy was preferred this could have been achieved just as easily by using the following command: -

./core-service=management/security-realm=LDAPRealm/authentication=ldap/cache=by-search-time:add(eviction-time=300, cache-failures=true, max-cache-size=100)

I am not going to go through all of the other places in the model where caches can be enable but for the remaining two areas the command would be the same with just a modification to the address.

Cache Operations

Once caching is enabled the caches run fairly independently, however there are a couple of additional tasks you may want to perform yourself.  These examples are using the cache I defined above, however with address modifications they are applicable to the additional caches that can be defined.

Inspect The Cache Size

If you want to see how many entries are cached the following command can be executed to include runtime attributes: -

./core-service=management/security-realm=LDAPRealm/authentication=ldap/cache=by-access-time:read-resource(include-runtime=true)

{

    "outcome" => "success",

    "result" => {

        "cache-failures" => true,

        "cache-size" => 1,

        "eviction-time" => 300,

        "max-cache-size" => 100

    }

}

This was executed after I authenticated as one user so you can see one entry is in the cache.

Test The Cache Contents

If you want to check if the cache contains a reference to a specific user a command can be executed as follows: -

./core-service=management/security-realm=LDAPRealm/authentication=ldap/cache=by-access-time:contains(name=TestUserOne)

{

    "outcome" => "success",

    "result" => true

}

In this case TestUserOne is the user that I authenticated as.

Flushing The Cache

Finally it is possible to flush items from the cache, either the whole cache can be flushed or individual items can be flushed e.g.

./core-service=management/security-realm=LDAPRealm/authentication=ldap/cache=by-access-time:flush-cache(name=TestUserOne)

In this case the specific entry was flushed, had the name parameter been omitted from the operation the whole cache would have been flushed.

XML Configuration

That concludes this blog post, to complete this example here is the resulting XML for my security realm definition with the cache now defined: - 

<security-realm name="LDAPRealm">

    <authentication>

        <ldap connection="MyLdapConnection" 

              base-dn="ou=users,dc=group-to-principal,dc=wildfly,dc=org">

            <cache type="by-access-time" eviction-time="300" 

                   cache-failures="true" max-cache-size="100"/>

            <username-filter attribute="uid"/>

        </ldap>

    </authentication>

    <authorization>

        <ldap connection="MyLdapConnection">

            <username-to-dn>

               <username-filter attribute="uid"

                    base-dn="ou=users,dc=group-to-principal,dc=wildfly,dc=org" />

            </username-to-dn>

            <group-search group-name="SIMPLE" iterative="true" 

                          group-name-attribute="uid">

                <group-to-principal search-by="DISTINGUISHED_NAME" 

                     base-dn="ou=groups,dc=group-to-principal,dc=wildfly,dc=org" 

                     prefer-original-connection="true">

                    <membership-filter principal-attribute="uniqueMember"/>

                </group-to-principal>

            </group-search>

        </ldap>

    </authorization>

</security-realm>

WildFly Command Line Interface - Connection Aliases

A while ago now I added a new feature to the WildFly command line interface to add support for aliases to be defined for connections to servers - I realise however that sometimes when it comes to the CLI capabilities such as this are not as visible until someone actually points out that they exist.

Note: This blog post is written based on WildFly 8.1.0.Final, however this feature was added for WildFly 8.0.0.Final under the following issue WFLY-1850.  This feature is also applicable to later versions of WildFly although most likely using a later version of the schema for configuration.

Configuration File

The first thing that may be new to you is that the command line interface is actually configurable, out of the box we include a base configuration and default settings that are compatible with the application server it is bundled with.

By default the file is called jboss-cli.xml and as of WildFly 8.1.0.Final it is using the following namespace urn:jboss:cli:2.0.

When the CLI starts up a sequence of locations are checked for the location of the configuration file: -

1. If the system property jboss.cli.config is set it is used as the name of the file, this can either be an absolute path or relative to the CLI's working dir.
2. The working dir is checked for a file called jboss-cli.xml.
3. The bin folder of the WildFly installation is checked for a file called jboss-cli.xml.

Failing that a default configuration is assumed not based on any file configuration.

Defining Aliases

The default configuration already contains an example of an alias definition so to define an alias first uncomment this block: -

    <!-- Example controller alias named 'Test'
    <controllers>
        <controller name="Test">
            <protocol>http-remoting</protocol>
            <host>localhost</host>
            <port>9990</port>
        </controller>
    </controllers>
    -->

For each alias definition four pieces of information can be defined: -

1. A unique name.
2. The protocol.
3. The host name.
4. The port.

Connecting By Alias

Using an alias is no more complicated than if you were already connecting to servers and specifying a host name.

On starting the CLI you can connect using an alias with a command similar to: -

./jboss-cli.sh -c --controller=Test

Alternatively start the CLI without any arguments and specify the name of the alias when you call connect e.g.

[disconnected /] connect Test

Why Use Aliases

The next question is why would you want to use aliases.

In the first case if you are administering a number of servers as an example you could have development, staging and production by defining aliases you no longer need to remember the address to use to connect to each of the servers - in addition to that should the address of any of these servers be modified in the future you can update the alias definition and continue to use the command you are used to for connecting to that server.

Secondly and one point I did not mention above, the name of the alias could be an address. So say you have a bunch of scripts that always call connect localhost:9999 - by default the server no longer listens on port 9999, an alias can be used to map this to the correct address.