WildFly Elytron - Credential Store - Next Steps

During the development of WildFly 11 where we introduced the WildFly Elytron project to the application server one of the new features we added was the credential store.

https://developers.redhat.com/blog/2017/12/14/new-jboss-eap-7-1-credential-store/

Now that we are planning the next stages of development for WildFly 15 and 16 we are revisiting some of the next steps for the credential store, this blog post explores some of the history of the current decisions made with the credential store and the types of enhancement being requested to develop this further.

Anyone making use of either the CredentialStore or Vault is encouraged to provider your feedback so we can take this into account as the next stages are planned.

Key Differences To Vault

Prior to the credential store the PicketBox vault was the solution used for the encryption of credentials and other fields within the application server's configuration, the credential store approach was dedicated to the secure storage of credentials.

Where the PicketBox vault is used in the application server a single Vault is defined across the whole server and aliases from the vault are referenced via expressions in the server's configuration which allows for the values to be retrieved in clear text.

The credential store on the other hand allows for multiple credential store instances to be defined, resources that make use of the credentials have been updated to a special attribute type where both the name of the credential store can be specified and the alias of the credential within the credential store.

The credential store resources additionally support management operations to allow for entries within the credential store to be manipulated either by adding / removing entries or by updating existing entries.

Uses of the Store

Reviewing where the credential store is used we seem to have two predominant scenarios.

Unlocking Local Resources

In this case a credential is required to unlock a local resource such as a KeyStore, there is no remote authentication to be performed and a credential is generally only required for decrypting the contents of the store.  This is the simplest use of the store and once the resource is unlocked it is not likely to need unlocking again.

Accessing Remote Resources

The second use we see is for services accessing a remote resource, in this case the credentials for the connection are obtained from the credential store.

This scenario has a reasonable amount of history also attached to the current implementation, it tended to be the case that if you were to access a remote resource it would either not be secured or it would be secured and authentication would require a username and password.  Additionally any SSL related configuration would be handled completely independently.

In recent years however there has been a greater demand for alternative authentication mechanisms, there has been a lot of demand for Kerberos both with the server being given it's own account for authentication and also for the propagation of a remote user authenticated against the server using Kerberos.  We haven't seen requests yet for the application server but I suspect OAuth scenarios will be requested soon.

In both of these cases the security has moved from username / password authentication to more advanced scenarios.

In parallel to the credential store WildFly Elytron has also introduced an Authentication Client API, this API can be used for configuring the client side authentication policies for various mechanisms, including scenarios that support propagation of the current identity.  The authentication client configuration also allows SSLContext configurations to be associated with a specific destination.

This now raises the question, should services establishing a remote connection which requires authentication and SSL configuration now reference an authentication client configuration instead of the current assumption that a username and credential reference is sufficient?

This question becomes quite important as it affects the approach we take to a lot of the enhancements requested of us quite significantly.

Next Features

The features currently being requested against the credential store generally cover three broad areas: -

1. Automation of updates to the store.
2. Real time updates.
3. Support for expressions.

Automation of Updates to the Store

The general motivation for this enhancement is to simplify the steps configuring resources which require credentials, using the PicketBox Vault implementation the vault would need to be manipulated offline using a command line tool and then references from the application server configuration.

The CredentialStore has already moved on from this partially as management operations have been exposed to allow the contents of the store to be directly manipulated using management requests so the store can be manipulated directly from the management tools.  However an administrator is still required to operate on the two resources completely independently.

We predominantly have two options to simplify this further.

We could take the decision that further enhancement is now a tooling issue, the admin clients could detect a resource is being added that supports credential store references or the password on an existing resource that supports credential store references is being set and provide guidance / automation to persist the credential in the credential store.

1. Would you like to store this credential in a credential store?
2. Which credential store would you like to use? Or would you like to create a new one?
3. What alias would you like the credential to be stored under?

Alternatively, the credential reference attribute supports specifying a reference to a credential store and an alias in the credential store - this attribute however also supports specifying a clear text password.  We could automate the manipulation in the management tier and if a clear text password is specified on a resource referencing a credential store automatically add it to the credential store and remove it from the model - if no alias is specified automatically generate one.

We could also support a combination of the two approaches as in the management tier although we could support the interception of a new clear password if a store needs to be created that would be more involved than we could automate.

Real Time Updates

Where new credentials are added to a credential store using management operations they are already available for use immediately in the application server process without a restart.  However we still have some areas to consider further real time update support: -

1. Updates to the store on the filesystem.
2. Complete replacement of an existing store.
3. Updates to credentials using management operations.

In these three cases the primary interest is credentials which are already in use in the application server, however in the case of #1 and #2 it could relate to the addition of new credentials to be used immediately.

One point to consider is although our default credential store implementation is file based custom implementations could be in use which are not making use of any local files.

At the credential store level we likely should consider various modes to detect changes when emitting notifications: -

1. File system monitoring.
2. Notifications from within the implementation of the store.
3. Administrator triggered reloads of either the full store or individual aliases.

At the service level where a service is making use of a credential it is likely we would want to decide how to handle updates on a service by service basis.  It is unlikely we would want to automatically restart / replace services as for some services which make use of credentials this could cause a cascade of service restarts potentially leading the redeployment of deployments.

I expect some form of notifications will be required, at the coarsest level notifications could be emitted for all services accessing a specific credential store - this however could trigger a significant overhead as a single store could contain a large number of entries used across a large number of resources.  Instead we could emit notifications just to the resources using the affected aliases, this would be more efficient from the perspective of the notifications but the complexity now is where a coarse update has been updates to a store such as the underlying file being replaced or a full store refresh being triggered by an administrator we now need to identify which credentials were really modified.

Support for Expressions

It was a deliberate decision to move away from using expressions, however there are still some demands for expressions that need to be considered.

Overall the design decisions within WildFly Elytron have always considered a desire to move away from clear text passwords being present in the application server process, where expressions are used the only route available is to obtain the clear text representation of a String and pass it around the related services.

One of the enhancements delivered for WildFly 11 was to support multiple credential stores concurrently within the application server, by moving to the complex attribute we were able to make use of capability references to select which credential store to use with a second attribute selecting the alias in the store.

Another consideration was the desire for automatic updates to be applied via the management tier, by moving from expressions to a complex attribute it opens up the options to intercept these values and persist them in the store.

A further (and possibly the greatest) consideration was the desired support for automatic updates to credentials currently in use in the server, by moving to the capability references aided by the complex attribute definition services can now obtain a direct reference to the store instead of having a credential automatically resolved.  By having a reference to a credential store we can potentially add support for direct notifications of updates applied to that store.  Where a credential is updated different services may want to respond in different ways, this is why a reference is needed.  Within the management tier we can not silently automate the updates, if we were to do so it would likely involve the removal and replacement of the service which could have a side effect of restarting many other services including the deployments.

The biggest restriction of not supporting expressions is attributes for anything other than a credential can no longer be loaded from the credential store - but the missing piece of information is how that is really used and why?

As an example usernames could be loaded using expressions, is this because in some environments the username is being considered as sensitive as the credential?  Or is it the case that where a credential is loaded from a store it is easier to load the username from the same store.

If the answer is co-location of the username and password then a more suitable path to look into may be the externalisation of the authentication client configuration allowing the complete client authentication policy to be handled as a single unit.

If we are still left with attributes in the configuration that need to be stored securely the next question is do they strictly need to be removed from the configuration and looked up from the store?  An alternative option we have to consider is supporting the encryption / decryption of Strings inlined in the management model using a credential from the store.

Other Enhancements

Following on from the main enhancements listed above there is also a set of additional enhancements we could consider.

Injection / Credential Store Access for Deployments

Deployments can already access the authentication client configuration, however if access to raw credentials is required it may help to access the store - this could additionally mean a deployment could manipulate the store.

Permission Checks

Once accessed within deployments, making use of the SecurityIdentity permission checks we could perform permission checks for different read / write operations on the credential store.

Auditing

The credential store could be updated to emit security events as it is accessed, by emitting security events these can be output to audit logs or sent to other event analysing frameworks.