Thursday, 27 November 2014

WildFly 9 - Kerberos Authentication with Remoting - Part 1

This blog post is in relation to a new feature added to WildFly 9 under WFCORE-106, however it is not currently included in a release so for now you will need to build WildFly yourself or use one of the nightly builds.

If you are not familiar with building WildFly have a look at HackingOnWildFly.

If you want to access a nightly build they are available from WildFly-latest-master.


This blog post is the first part of a two part article on how to enable Keberos authentication for Remoting within WildFly 9, in Part 1 I discuss how to enable Keberos authentication for use with the CLI for managing the application server which makes use of JBoss Remoting to communicate with the server.

In Part 2 I will be looking at how to take an existing secured EJB deployment from within the quickstarts and enable Kerberos based authentication.

In a previous blog post WildFly 9 - Kerberos Authentication for Domain Management Over HTTP I have already described a lot of the steps required to enable Kerberos authentication for HTTP, for the steps that are also applicable for authentication with Remoting I will briefly mention them but will not duplicate the descriptions here.

Service Principals

One important aspect of Kerberos based authentication is that the service requiring authentication needs to be represented by it's own service principal, the naming of this principal is in the form '{service_name}/{hostname}' - where HTTP servers are secured using Kerberos the service_name is 'HTTP' - however for Remoting the service_name is expected to be 'remote'.

My Environment

My test environment that I am using for the basis of this blog post is the same as I described previously, the main difference here is that now I have a service principal called remote/ which is in a keytab called remote.keytab.

As before any example commands will need to be updated to reflect your own environment.

Configuration Updates

As with the previous blog post system properties can be set as an easy way to define the realm name and the address of the KDC, after that similar updates also need to be made to the ManagementRealm: -

  1. Add a server identity for Kerberos.
  2. Add the keytab for the remote service principal.
  3. Add kerberos to the authentication resource of the security domain.
  4. Reload the server.
Steps 1, 3, and 4 are identical to those performed previously but in this example I would use the following command to add my remote keytab.

[ /] ./core-service=management/security-realm=ManagementRealm/server-identity=kerberos/keytab=remote\/, relative-to=jboss.server.config.dir, debug=true)
    "outcome" => "success",
    "response-headers" => {
        "operation-requires-reload" => true,
        "process-state" => "reload-required"

Running the CLI

As before I have used the kinit command to obtain my own ticket granting ticket from the KDC: -

[darranl@localhost bin]$ kinit darranl@DAL.WILDFLY.ORG
Password for darranl@DAL.WILDFLY.ORG: 
[darranl@localhost bin]$ klist
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: darranl@DAL.WILDFLY.ORG

Valid starting     Expires            Service principal
27/11/14 17:44:50  28/11/14 17:44:47  krbtgt/DAL.WILDFLY.ORG@DAL.WILDFLY.ORG
renew until 04/12/14 17:44:47

The next step is to start the CLI, as I start the CLI I set three system properties: -
The first two could be optional depending on local environmental settings but the third one is important as this allows the GSSAPI implementation to make use of the identity managed at the OS level.

In the case of my testing I am running the CLI on the same machine that I am running the application server so it is also important to ensure I disable local authentication otherwise Kerberos authentication will not get a chance to be used so I add the argument --no-local-auth.

This means the resulting command is: -

[darranl@localhost bin]$ ./ -c \
    --no-local-auth \ \ [ /]

I can execute the :whoami command to double check the user I am authenticated as.

[ /] :whoami
    "outcome" => "success",
    "result" => {"identity" => {
        "username" => "darranl",
        "realm" => "ManagementRealm"

Additional Config

The example above is making use of a lot of default behaviour within the CLI, if you would prefer to control the Kerberos identity further it is possible to override the JAAS security domain used internally by adding the following system property to point to your own JAAS configuration{config}, this configuration should contain a security domain called '' that contains a Krb5LoginModule definition for your own environment. One example of a reason you may want to do this is if your clients identity can also be obtained from a keytab.

Friday, 17 October 2014

WildFly 9 - Kerberos Authentication for Domain Management Over HTTP

This blog post is in relation to a new feature added to WildFly 9 under WFCORE-105, however it is not currently included in a release so for now you will need to build WildFly yourself or use one of the nightly builds.

If you are not familiar with building WildFly have a look at HackingOnWildFly.

If you want to access a nightly build they are available from WildFly-latest-master.


This blog post is specifically about the steps and configuration required to enable Kerberos based authentication for domain management over HTTP, this blog post does not describe how to define service principals and obtain the keytab or how to configure the web browser - there is already plenty of information out there covering those topics.

My Environment

The examples described in this blog post are all in the context of my test environment.

My KDC is service requests for the realm DAL.WILDFLY.ORG and the address of my KDC is

The HTTP management interface of my server installation is already listening on host and I have a keytab called web.keytab which contains the service principal HTTP/

Where you see any of these values in the following example obviously you will need to substitute in your own specific values for your own environment.

General Kerberos Settings

It is possible to rely on your local environments Kerberos configuration for things such as KDC identification and the resolution of the KDC's address, however for my own test environment where I am using a single KDC I prefer to just set a couple of system properties using the following CLI commands.

[ /] 
{"outcome" => "success"}
[ /] 
{"outcome" => "success"}

Enabling Kerberos Authentication

The steps to enable Kerberos authentication are split into two parts, first of all we need to define the servers own identity which is loaded from it's keytab and secondly once that is defined we need to enable the authentication.

Servers Identity

The first step is to now configure how the server will load it's own identity from a keytab, to do this I copy my web.keytab to the standalone/configuration folder of my installation and execute the following CLI commands: -

[ /] ./core-service=management/security-realm=ManagementRealm/server-identity=kerberos:add
    "outcome" => "success",
    "response-headers" => {
        "operation-requires-reload" => true,
        "process-state" => "reload-required"
[ /] ./core-service=management/security-realm=ManagementRealm/server-identity=kerberos/keytab=HTTP\/, relative-to=jboss.server.config.dir, debug=true)
    "outcome" => "success",
    "response-headers" => {
        "operation-requires-reload" => true,
        "process-state" => "reload-required"

The first command adds a container resource to the security realm to say that we will be adding a Kerberos identity, the subsequent command adds a reference to the keytab.  take note of where the principal name is specified within the address for the keytab and see how it needed to be escaped, also for this keytab I have enabled some additional debug output for when we communicate with the KDC.

Also note that these commands have put the server in the reload-required state.

It is possible that a single server could be accessible using multiple host names, for this reason it is possible to add multiple keytabs to the security realm,  the keytab definitions also have a for-hosts attribute that can be used to specify which keytab should be used for which host, failing that there will be an attempt to match the host name to the principal name.

In general we could proceed to the next section but it worth now executing :reload so that we can test our new configuration before enabling the actual authentication.

[ /] :reload                                  
    "outcome" => "success",
    "result" => undefined

The keytab resource then has a test operation that can be invoked to verify the identity can be obtained.

[ /] ./core-service=management/security-realm=ManagementRealm/server-identity=kerberos/keytab=HTTP\/
    "outcome" => "success",
    "result" => {"subject" => "Subject:
Principal: HTTP/
Private Credential: /home/darranl/src/wildfly9/wildfly/dist/target/wildfly-9.0.0.Alpha2-SNAPSHOT/standalone/configuration/web.keytab
Private Credential: Kerberos Principal HTTP/ Version 1key EncryptionKey: keyType=17 keyBytes (hex dump)=
0000: 8C D1 9C 24 86 84 D4 25   A3 12 FF DE A9 0D 9B 26  ...$...%.......&

At this point it is absolutely critical that you verify in the above output that one of the Private Credentials is actually a 'Kerberos Principal' and not just the reference to the keytab location.

Switch on Authentication

To switch on authentication a simple addition to the realm is required: -

[ /] ./core-service=management/security-realm=ManagementRealm/authentication=kerberos:add
    "outcome" => "success",
    "response-headers" => {
        "operation-requires-reload" => true,
        "process-state" => "reload-required"

At this point :reload needs to be executed again and now I can try and connect using my web browser.

Load the Admin Console

Firstly I make sure I have obtained my own ticket granting ticket from the KDC: -

[darranl@localhost tmp]$ kinit darranl@DAL.WILDFLY.ORG
Password for darranl@DAL.WILDFLY.ORG: 
[darranl@localhost tmp]$ klist
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: darranl@DAL.WILDFLY.ORG

Valid starting     Expires            Service principal
17/10/14 13:08:18  18/10/14 13:08:15  krbtgt/DAL.WILDFLY.ORG@DAL.WILDFLY.ORG
renew until 24/10/14 13:08:15

As the security realm is based on the default configuration username/password based authentication is still enabled, for this reason you will need to ensure at least one user is defined by running the add-user script.

Now I load my web browser and attempt to connect using the admin console.

With no additional password prompt I am now authenticated as the user I previously obtained the ticket granting ticket for.

It is worth pointing out that my identity here includes the name of the realm, if desired when I added the authentication=kerberos to the realm I could have added the attribute remove-realm=true to strip the realm name from the username.

Finally if my web browser had not been enabled for Kerberos authentication or if I had not already obtained the ticket granting ticket I would have been prompted for a username and password as normal.

XML Configuration

Finally all of this configuration could have been defined manually in the application server configuration, the following is the resulting realm definition after the above CLI commands have been executed.

<security-realm name="ManagementRealm">
            <keytab principal="HTTP/" path="web.keytab" relative-to="jboss.server.config.dir" debug="true"/>
        <local default-user="$local" skip-group-loading="true"/>
        <properties path="" relative-to="jboss.server.config.dir"/>
    <authorization map-groups-to-roles="false">
        <properties path="" relative-to="jboss.server.config.dir"/>

Next Steps

The next steps now are to enable Kerberos authentication for our native interface so that clients such as the CLI and JMX Console can also take advantage of Kerberos based authentication.

Thursday, 11 September 2014

Kerberos EncryptedData NULL key / keytype mismatch.

For anyone who has ever been in the position of debugging Kerberos interoperability issues where Java is concerned I am fairly sure you would have come across the situation where you are attempting to search for some meaning in a cryptic error message you have received just to stumble on message posts from other users with the same error but no explanation as to the cause or a solution.

The purpose of this blog post is to describe one such error that I am currently investigating, I will say upfront however I have got to the bottom of why the error is being reported and have a possible workaround for some situations but the quest for a solution is ongoing.

If and when a complete solution is available I will update this blog post with the additional information.

In this case I have a JBoss EAP server handling incoming SPNEGO messages from different clients.

The Errors

Before describing the findings here are a couple of the errors I have been seeing: -

Caused by: KrbException: EncryptedData is encrypted using keytype NULL but decryption key is of type AES256 CTS mode with HMAC SHA1-96
at [rt.jar:1.7.0_17]
at<init>( [rt.jar:1.7.0_17]
at$OverloadedChecksum.<init>( [rt.jar:1.7.0_17]
at<init>( [rt.jar:1.7.0_17]
at [rt.jar:1.7.0_17]


Caused by: KrbException: EncryptedData is encrypted using keytype DES3 CBC mode with SHA1-KD but decryption key is of type NULL
at [rt.jar:1.7.0_17]
at<init>( [rt.jar:1.7.0_17]
at$OverloadedChecksum.<init>( [rt.jar:1.7.0_17]
at<init>( [rt.jar:1.7.0_17]
at [rt.jar:1.7.0_17]
... 35 more

For the purpose of identifying if your error message is related to this it is absolutely essential that you also compare the stack trace to ensure it is similar, i.e. the line numbers may be different but the classes in the stack should be almost identical.

The first of these error messages was reported when connecting to a server using Firefox from a Mac OS client, the second was reported using Firefox on a Fedora 20 client.

The Trigger

Within the SPNEGO authentication a Kerberos AP_REQ message is sent from the client to the server, within this message it is possible for the client to embed their own credential so that it can be delegated to the server.  For both of the error message above the KrbCred creation, EncryptedData decryption and the resulting validation and error are only applicable if the incoming message does contain a delegated credential.

When it comes to credential delegation and Kerberos this is something that the client decides it is going to do, the server has not actually requested that the client does delegate their credentials.

The Workaround

As this error is only encountered where credentials are delegated to the server the most simple workaround is to disable credential delegation in your client.  That way the AP_REQ message will not contain delegated credentials and the portion of the stack above reporting the error message will not be hit.

Unfortunately this workaround is only suitable if your server side service is not making use of delegated credentials but many services use SPNEGO without requiring delgation.

The Trigger II

So what is actually happening during the parsing that is triggering the error.

Within the Kerberos protocol a session key is established, certain fields can be encrypted using the session key and on parsing the message this session key can be used to decrypt the field.  

The encoded representation of an encrypted attribute contains a field that specifies the encryption type in use, for the above errors we see three types mentioned aes256-cts-hmac-sha1-96, des3-cbc-sha1, and also NULL.  Null is a special case as it specified that an encrypted field is not actually encrypted.

So back to the Java implementation, before parsing and possibly decrypting the delegated credential the Java implementation checks the type of the agreed session key.  If the session key is for aes256-cts-hmac-sha1-96, aes128-cts-hmac-sha1-96, or is arcfour then an assumption is made that the session key should be used to decrypt the credential.  For all other cases it assumes the encryption type will be NULL and that a NULL key should be used for the decryption.

So for the first error I posted above the Mac OS client used NULL for encrypting the delegated credential however as the session key is aes256-cts-hmac-sha1-96 the parsing is expecting this to have been used.

For the second error with the Linux client the session key is now des3-cbc-sha1, the Linux client did encrypt the delegated credential but the Java implementation has made the assumption it should not be doing that.

The Next Steps

As I said at the start at the time of posting this I do not have the solution, only the information that will hopefully make it easier for you to see what the error means and one possible workaround.

Overall this error is as a result of different approaches to message creation and parsing between the different clients and the Java implementation - so now I need to get to the bottom of which side is actually at fault.  My initial suspicion is the Java side as it should be possibly to identify the encryption type that was used before this error is reported but that is going to require some further investigation and discussions.

Thursday, 28 August 2014

WildFly 8 Security Realms - LDAP Caching

Another feature added to WildFly 8 that you may not be aware of unless you have gone looking for it is the ability to define caches on security realms where LDAP is being used for authentication or group loading.  This was added under WFLY-1523.

If you are familiar with the caching already in place for JAAS based security domains please be aware that the approach taken for the security realms is very different, for JAAS the key to the cache is essentially using the users supplied username and password and the cached value is the entire interaction with LDAP including all of the groups recursively found.  Within the security realms the caching is much more fine grained and it is the results for individual queries which are cached independently rather than the overall result cached as one.  This will be explained further in this post, for now this essentially means that it is possible that the results from queries for one user can be re-used for another user - this would occur where iteratively querying the group membership information of groups.

Base Configuration

Before describing how to enable caching it makes sense to start with an existing security realm already configured to authenticate users against LDAP and load their group membership information from LDAP, the following is one such possible configuration: -

    "core-service" : {
        "management" : {
            "security-realm" : {
                "LDAPRealm" : {
                    "authentication" : {"ldap" : {
                        "allow-empty-passwords" : false,
                        "base-dn" : "...",
                        "connection" : "MyLdapConnection",
                        "recursive" : false,
                        "user-dn" : "dn",
                        "username-attribute" : "uid",
                        "cache" : null
                    "authorization" : {"ldap" : {
                        "connection" : "MyLdapConnection",
                        "group-search" : {"group-to-principal" : {
                            "base-dn" : "...",
                            "group-dn-attribute" : "dn",
                            "group-name" : "SIMPLE",
                            "group-name-attribute" : "uid",
                            "iterative" : true,
                          "principal-attribute" : "uniqueMember",
                            "search-by" : "DISTINGUISHED_NAME",
                            "cache" : null
                        "username-to-dn" : {"username-filter" : {
                            "attribute" : "uid",
                            "base-dn" : "...",
                            "force" : false,
                            "recursive" : false,
                            "user-dn-attribute" : "dn",
                            "cache" : null

This example has been cleaned up slightly but the main point to note is that this configuration has three key areas.

  • Authentication
If you are already using LDAP with security realms this should be the most familiar, effectively during authentication we discover the users distinguished name using this definition and attempt to connect to LDAP using their supplied credential to verify they are who they claim to be.
  • A username-to-dn definition in group search
When it comes to group searching we may rely on the availability of the users distinguished name, this block is not used in all situations but essentially it is a second attempt to discover a users distinguished name, this is more likely to be required if a second form of authentication was supported e.g. local authentication.
  • A group-to-principal group search

Then finally there is the group search definition, in this case it is an iterative search - what that means is that first all of the groups will be identified that the user is directly a member of - after that a search will be performed for each of those groups to identify the groups that those groups are a member of, this process continues until either a cyclic reference is detected or the final groups are not members of any further groups.

Before moving onto the next section please note in the above example there are three points that have a sub resource of "cache" : null , it is at each of these points that a cache can be defined to cache the LDAP interaction for that portion of the model.

The Caches

When enabling a cache it is possible to chose from one of two eviction strategies, the first being by access time and the second by search time.  If you choose by access time then the item in the cache will be evicted from the cache after a certain time period has elapsed since it was last used, if you choose by search time it will be evicted after the configured time has elapsed since the item was added to the cache regardless of if it has been accessed later.

In addition to choosing the eviction strategy the following attributes can also be set on a cache: -

  • eviction-time
This one should be fairly self explanatory and is the time in seconds used for evictions depending on the chosen strategy. 
  • cache-failures
This one is more important for the cache used in the authentication section and controls if the results of failed LDAP searches are cached to prevent the LDAP server from being hit again - this risk here is that without the next attribute a remote user could fill up the cache by trying many different users that do not exist.
  • max-cache-size
In addition to the time based strategy it is also possible to define a maximum size for the cache, if the maximum size of the cache is reached the oldest item in the cache will be evicted to make room for a new item being added - this is the final level of protection to prevent a cache that is caching failures from using up all of the available heap space to cache those failures.

Enabling The Cache 

So that is the description of the different configuration choices you can make, the following is the CLI command you can use to add a cache for the authentication related LDAP access: -

./core-service=management/security-realm=LDAPRealm/authentication=ldap/cache=by-access-time:add(eviction-time=300, cache-failures=true, max-cache-size=100)
This command enabled a cache that caches by access time with items automatically evicted after 5 minutes unless the maximum capacity of 100 items is reached - in addition to that failures are cached so the LDAP server will not be hit repeatedly for authentication attempts where a user is using an invalid user ID.

If the eviction by search time strategy was preferred this could have been achieved just as easily by using the following command: -
./core-service=management/security-realm=LDAPRealm/authentication=ldap/cache=by-search-time:add(eviction-time=300, cache-failures=true, max-cache-size=100)
I am not going to go through all of the other places in the model where caches can be enable but for the remaining two areas the command would be the same with just a modification to the address.

Cache Operations

Once caching is enabled the caches run fairly independently, however there are a couple of additional tasks you may want to perform yourself.  These examples are using the cache I defined above, however with address modifications they are applicable to the additional caches that can be defined.

Inspect The Cache Size

If you want to see how many entries are cached the following command can be executed to include runtime attributes: -

    "outcome" => "success",
    "result" => {
        "cache-failures" => true,
        "cache-size" => 1,
        "eviction-time" => 300,
        "max-cache-size" => 100
This was executed after I authenticated as one user so you can see one entry is in the cache.

Test The Cache Contents

If you want to check if the cache contains a reference to a specific user a command can be executed as follows: -
    "outcome" => "success",
    "result" => true
In this case TestUserOne is the user that I authenticated as.

Flushing The Cache

Finally it is possible to flush items from the cache, either the whole cache can be flushed or individual items can be flushed e.g.
In this case the specific entry was flushed, had the name parameter been omitted from the operation the whole cache would have been flushed.

XML Configuration

That concludes this blog post, to complete this example here is the resulting XML for my security realm definition with the cache now defined: - 

<security-realm name="LDAPRealm">
        <ldap connection="MyLdapConnection" 
            <cache type="by-access-time" eviction-time="300" 
                   cache-failures="true" max-cache-size="100"/>
            <username-filter attribute="uid"/>
        <ldap connection="MyLdapConnection">
               <username-filter attribute="uid"
                    base-dn="ou=users,dc=group-to-principal,dc=wildfly,dc=org" />
            <group-search group-name="SIMPLE" iterative="true" 
                <group-to-principal search-by="DISTINGUISHED_NAME" 
                    <membership-filter principal-attribute="uniqueMember"/>

Thursday, 21 August 2014

WildFly Command Line Interface - Connection Aliases

A while ago now I added a new feature to the WildFly command line interface to add support for aliases to be defined for connections to servers - I realise however that sometimes when it comes to the CLI capabilities such as this are not as visible until someone actually points out that they exist.

Note: This blog post is written based on WildFly 8.1.0.Final, however this feature was added for WildFly 8.0.0.Final under the following issue WFLY-1850.  This feature is also applicable to later versions of WildFly although most likely using a later version of the schema for configuration.

Configuration File

The first thing that may be new to you is that the command line interface is actually configurable, out of the box we include a base configuration and default settings that are compatible with the application server it is bundled with.

By default the file is called jboss-cli.xml and as of WildFly 8.1.0.Final it is using the following namespace urn:jboss:cli:2.0.

When the CLI starts up a sequence of locations are checked for the location of the configuration file: -

  1. If the system property jboss.cli.config is set it is used as the name of the file, this can either be an absolute path or relative to the CLI's working dir.
  2. The working dir is checked for a file called jboss-cli.xml.
  3. The bin folder of the WildFly installation is checked for a file called jboss-cli.xml.
Failing that a default configuration is assumed not based on any file configuration.

Defining Aliases

The default configuration already contains an example of an alias definition so to define an alias first uncomment this block: -

    <!-- Example controller alias named 'Test'
        <controller name="Test">

For each alias definition four pieces of information can be defined: -

  1. A unique name.
  2. The protocol.
  3. The host name.
  4. The port.

Connecting By Alias

Using an alias is no more complicated than if you were already connecting to servers and specifying a host name.

On starting the CLI you can connect using an alias with a command similar to: -

./ -c --controller=Test

Alternatively start the CLI without any arguments and specify the name of the alias when you call connect e.g.

[disconnected /] connect Test

Why Use Aliases

The next question is why would you want to use aliases.

In the first case if you are administering a number of servers as an example you could have development, staging and production by defining aliases you no longer need to remember the address to use to connect to each of the servers - in addition to that should the address of any of these servers be modified in the future you can update the alias definition and continue to use the command you are used to for connecting to that server.

Secondly and one point I did not mention above, the name of the alias could be an address. So say you have a bunch of scripts that always call connect localhost:9999 - by default the server no longer listens on port 9999, an alias can be used to map this to the correct address.