openstack/openstack-manuals
Code Issues Proposed changes

Add Fernet FAQ

Browse Source 
We've been getting various questions on Fernet tokens for a while. We should
add some of the common ones to docs.

Change-Id: I8dd42ef1657cf6312636f1f59a5897b6956a4ee4
This commit is contained in:
Lance Bragstad 2015-11-12 14:50:11 +00:00
parent da89b3b1cf
commit d0003aa133
3 changed files with 333 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
doc/admin-guide-cloud/source/identity_management.rst
View File

@ -21,6 +21,7 @@ command-line client.
keystone_integrate_with_ldap.rst
keystone_tokens.rst
keystone_token-binding.rst
keystone_fernet_token_faq.rst
keystone_use_trusts.rst
keystone_caching_layer.rst
identity_user_crud.rst

325
doc/admin-guide-cloud/source/keystone_fernet_token_faq.rst Normal file
View File

@ -0,0 +1,325 @@
===================================
fernet - Frequently Asked Questions
===================================
The following questions have been asked periodically since the initial release
of the fernet token format in Kilo.
What are the different types of keys?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A key repository is required by keystone in order to create fernet tokens.
These keys are used to encrypt and decrypt the information that makes up the
payload of the token. Each key in the repository can have one of three states.
The state of the key determines how keystone uses a key with fernet tokens. The
different types are as follows:
Primary key:
There is only ever one primary key in a key repository. The primary key is
allowed to encrypt and decrypt tokens. This key is always named as the
highest index in the repository.
Secondary key:
A secondary key was at one point a primary key, but has been demoted in place
of another primary key. It is only allowed to decrypt tokens. Since it was
the primary at some point in time, its existence in the key repository is
justified. Keystone needs to be able to decrypt tokens that were created with
old primary keys.
Staged key:
The staged key is a special key that shares some similarities with secondary
keys. There can only ever be one staged key in a repository and it must
exist. Just like secondary keys, staged keys have the ability to decrypt
tokens. Unlike secondary keys, staged keys have never been a primary key. In
fact, they are opposites since the staged key will always be the next primary
key. This helps clarify the name because they are the next key staged to be
the primary key. This key is always named as ``0`` in the key repository.
So, how does a staged key help me and why do I care about it?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The fernet keys have a natural lifecycle. Each key starts as a staged key, is
promoted to be the primary key, and then demoted to be a secondary key. New
tokens can only be encrypted with a primary key. Secondary and staged keys are
never used to encrypt token. The staged key is a special key given the order of
events and the attributes of each type of key. The staged key is the only key
in the repository that has not had a chance to encrypt any tokens yet, but it
is still allowed to decrypt tokens. As an operator, this gives you the chance
to perform a key rotation on one keystone node, and distribute the new key set
over a span of time. This does not require the distribution to take place in an
ultra short period of time. Tokens encrypted with a primary key can be
decrypted, and validated, on other nodes where that key is still staged.
Where do I put my key repository?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The key repository is specified using the ``key_repository`` option in the
keystone configuration file. The keystone process should be able to read and
write to this location but it should be kept secret otherwise. Currently,
keystone only supports file-backed key repositories.
..code-block:: ini
[fernet_tokens]
key_repository = /etc/keystone/fernet-keys/
What is the recommended way to rotate and distribute keys?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``keystone-manage`` command line utility includes a key rotation mechanism.
This mechanism will initialize and rotate keys but does not make an effort to
distribute keys across keystone nodes. The distribution of keys across a
keystone deployment is best handled through configuration management tooling.
Use ``keystone-manage fernet_rotate`` to rotate the key repository.
Do fernet tokens still expires?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Yes, fernet tokens can expire just like any other keystone token formats.
Why should I choose fernet tokens over UUID tokens?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Even though fernet tokens operate very similarly to UUID tokens, they do not
require persistence. The keystone token database no longer suffers bloat as a
side effect of authentication. Pruning expired tokens from the token database
is no longer required when using fernet tokens. Because fernet tokens do not
require persistence, they do not have to be replicated. As long as each
keystone node shares the same key repository, fernet tokens can be created and
validated instantly across nodes.
Why should I choose fernet tokens over PKI or PKIZ tokens?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The arguments for using fernet over PKI and PKIZ remain the same as UUID, in
addition to the fact that fernet tokens are much smaller than PKI and PKIZ
tokens. PKI and PKIZ tokens still require persistent storage and can sometimes
cause issues due to their size. This issue is mitigated when switching to
fernet because fernet tokens are kept under a 250 byte limit. PKI and PKIZ
tokens typically exceed 1600 bytes in length. The length of a PKI or PKIZ token
is dependent on the size of the deployment. Bigger service catalogs will result
in longer token lengths. This pattern does not exist with fernet tokens because
the contents of the encrypted payload is kept to a minimum.
Should I rotate and distribute keys from the same keystone node every rotation?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
No, but the relationship between rotation and distribution should be lock-step.
Once you rotate keys on one keystone node, the key repository from that node
should be distributed to the rest of the cluster. Once you confirm that each
node has the same key repository state, you could rotate and distribute from
any other node in the cluster.
If the rotation and distribution are not lock-step, a single keystone node in
the deployment will create tokens with a primary key that no other node has as
a staged key. This will cause tokens generated from one keystone node to fail
validation on other keystone nodes.
How do I add new keystone nodes to a deployment?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The keys used to create fernet tokens should be treated like super secret
configuration files, similar to an SSL secret key. Before a node is allowed to
join an existing cluster, issuing and validating tokens, it should have the
same key repository as the rest of the nodes in the cluster.
How should I approach key distribution?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Remember that key distribution is only required in multi-node keystone
deployments. If you only have one keystone node serving requests in your
deployment, key distribution is unnecessary.
Key distribution is a problem best approached from the deployment's current
configuration management system. Since not all deployments use the same
configuration management systems, it makes sense to explore options around what
is already available for managing keys, while keeping the secrecy of the keys
in mind. Many configuration management tools can leverage something like
``rsync`` to manage key distribution.
Key rotation is a single operation that promotes the current staged key to
primary, creates a new staged key, and prunes old secondary keys. It is easiest
to do this on a single node and verify the rotation took place properly before
distributing the key repository to the rest of the cluster. The concept behind
the staged key breaks the expectation that key rotation and key distribution
have to be done in a single step. With the staged key, we have time to inspect
the new key repository before syncing state with the rest of the cluster. Key
distribution should be an operation that can run in succession until it
succeeds. The following might help illustrate the isolation between key
rotation and key distribution.
1. Ensure all keystone nodes in the deployment have the same key repository.
2. Pick a keystone node in the cluster to rotate from.
3. Rotate keys.
a) Was is successful?
i) If no, investigate issues with the particular keystone node you
rotated keys on. Fernet keys are small and the operation for
rotation is trivial. There should not be much room for error in key
rotation. It is possible that the user does not have the ability to
write new keys to the key repository. Log output from
``keystone-manage fernet_rotate`` should give more information into
specific failures.
ii) If yes, you should see a new staged key. The old staged key should
be the new primary. Depending on the ``max_active_keys`` limit you
might have secondary keys that were pruned. At this point, the node
that you rotated on will be creating fernet tokens with a primary
key that all other nodes should have as the staged key. This is why
we checked the state of all key repositories in Step one. All other
nodes in the cluster should be able to decrypt tokens created with
the new primary key. At this point, we are ready to distribute the
new key set.
4. Distribute the new key repository.
a) Was it successful?
i) If yes, you should be able to confirm that all nodes in the cluster
have the same key repository that was introduced in Step 3. All
nodes in the cluster will be creating tokens with the primary key
that was promoted in Step 3. No further action is required until the
next schedule key rotation.
ii) If no, try distributing again. Remember that we already rotated the
repository and performing another rotation at this point will
result in tokens that cannot be validated across certain hosts.
Specifically, the hosts that did not get the latest key set. You
should be able to distribe keys until it is successful. If certain
nodes have issues syncing, it could be permission or network issues
and those should be resolved before subsequent rotations.
How long should I keep my keys around?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The fernet tokens that keystone creates are only secure as the keys creating
them. With staged keys the penalty of key rotation is low, allowing you to err
on the side of security and rotate weekly, daily, or even hourly. Ultimately,
this should be less time than it takes an attacker to break a ``AES256`` key
and a ``SHA256 HMAC``.
Is a fernet token still a bearer token?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Yes, and they follow exactly the same validation path as UUID tokens, with the
exception of being written to, and read from, a back end. If someone
compromises your fernet token, they have the power to do all the operations you
are allowed to do.
What if I need to revoke all my tokens?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To invalidate every token issued from keystone and start fresh, remove the
current key repository, create a new key set, and redistribute it to all nodes
in the cluster. This will render every token issued from keystone as invalid
regardless if the token has actually expired. When a client goes to
re-authenticate, the new token will have been created with a new fernet key.
What can an attacker do if they compromise a fernet key in my deployment?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If any key used in the key repository is compromised, an attacker will be able
to build their own tokens. If they know the ID of an administrator on a
project, they could generate administrator tokens for the project. They will be
able to generate their own tokens until the compromised key has been removed
from from the repository.
I rotated keys and now tokens are invalidating early, what did I do?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Using fernet tokens requires some awareness around token expiration and the key
lifecycle. You do not want to rotate so often that secondary keys are removed
that might still be needed to decrypt unexpired tokens. If this happens, you
will not be able to decrypt the token because the key the was used to encrypt
it is now gone. Only remove keys that you know are not being used to encrypt or
decrypt tokens.
For example, your token is valid for 24 hours and we want to rotate keys every
six hours. We will need to make sure tokens that were created at 08:00 AM on
Monday are still valid at 07:00 AM on Tuesday, assuming they were not
prematurely revoked. To accomplish this, we will want to make sure we set
``max_active_keys=6`` in our keystone configuration file. This will allow us to
hold all keys that might still be required to validate a previous token, but
keeps the key repository limited to only the keys that are needed.
The number of ``max_active_keys`` for a deployment can be determined by
dividing the token lifetime, in hours, by the frequency of rotation in hours
and adding two. Better illustrated as::
token_expiration = 24
rotation_frequency = 6
max_active_keys = (token_expiration / rotation_frequency) + 2
The reason for adding two additional keys to the count is to include the staged
key and a buffer key. This can be shown based on the previous example. We
initially setup the key repository at 6:00 AM on Monday, and the initial state
looks like::
$ ls -la /etc/keystone/fernet-keys/
drwx------ 2 keystone keystone 4096 .
drwxr-xr-x 3 keystone keystone 4096 ..
-rw------- 1 keystone keystone 44 0 (staged key)
-rw------- 1 keystone keystone 44 1 (primary key)
All tokens created after 6:00 AM are encrypted with key ``1``. At 12:00 PM we
will rotate keys again, resulting in::
$ ls -la /etc/keystone/fernet-keys/
drwx------ 2 keystone keystone 4096 .
drwxr-xr-x 3 keystone keystone 4096 ..
-rw------- 1 keystone keystone 44 0 (staged key)
-rw------- 1 keystone keystone 44 1 (secondary key)
-rw------- 1 keystone keystone 44 2 (primary key)
We are still able to validate tokens created between 6:00 - 11:59 AM because
the ``1`` key still exists as a secondary key. All tokens issued after 12:00 PM
will be encrypted with key ``2``. At 6:00 PM we do our next rotation, resulting
in::
$ ls -la /etc/keystone/fernet-keys/
drwx------ 2 keystone keystone 4096 .
drwxr-xr-x 3 keystone keystone 4096 ..
-rw------- 1 keystone keystone 44 0 (staged key)
-rw------- 1 keystone keystone 44 1 (secondary key)
-rw------- 1 keystone keystone 44 2 (secondary key)
-rw------- 1 keystone keystone 44 3 (primary key)
It is still possible to validate tokens issued from 6:00 AM - 5:59 PM because
keys ``1`` and ``2`` exist as secondary keys. Every token issued until 11:59 PM
will be encrypted with key ``3``, and at 12:00 AM we do our next rotation::
$ ls -la /etc/keystone/fernet-keys/
drwx------ 2 keystone keystone 4096 .
drwxr-xr-x 3 keystone keystone 4096 ..
-rw------- 1 keystone keystone 44 0 (staged key)
-rw------- 1 keystone keystone 44 1 (secondary key)
-rw------- 1 keystone keystone 44 2 (secondary key)
-rw------- 1 keystone keystone 44 3 (secondary key)
-rw------- 1 keystone keystone 44 4 (primary key)
Just like before, we can still validate tokens issued from 6:00 AM the previous
day until 5:59 AM today because keys ``1`` - ``4`` are present. At 6:00 AM,
tokens issued from the previous day will start to expire and we do our next
scheduled rotation::
$ ls -la /etc/keystone/fernet-keys/
drwx------ 2 keystone keystone 4096 .
drwxr-xr-x 3 keystone keystone 4096 ..
-rw------- 1 keystone keystone 44 0 (staged key)
-rw------- 1 keystone keystone 44 1 (secondary key)
-rw------- 1 keystone keystone 44 2 (secondary key)
-rw------- 1 keystone keystone 44 3 (secondary key)
-rw------- 1 keystone keystone 44 4 (secondary key)
-rw------- 1 keystone keystone 44 5 (primary key)
Tokens will naturally expire after 6:00 AM, but we will not be able to remove
key ``1`` until the next rotation because it encrypted all tokens from 6:00 AM
to 12:00 PM the day before. Once we do our next rotation, which is at 12:00 PM,
the ``1`` key will be pruned from the repository::
$ ls -la /etc/keystone/fernet-keys/
drwx------ 2 keystone keystone 4096 .
drwxr-xr-x 3 keystone keystone 4096 ..
-rw------- 1 keystone keystone 44 0 (staged key)
-rw------- 1 keystone keystone 44 2 (secondary key)
-rw------- 1 keystone keystone 44 3 (secondary key)
-rw------- 1 keystone keystone 44 4 (secondary key)
-rw------- 1 keystone keystone 44 5 (secondary key)
-rw------- 1 keystone keystone 44 6 (primary key)
If keystone were to receive a token that was created between 6:00 AM and 12:00
PM the day before, encrypted with the ``1`` key, it would not be valid because
it was already expired. This makes it possible for us to remove the ``1`` key
from the repository without negative validation side-effects.

13
doc/admin-guide-cloud/source/keystone_tokens.rst
View File

@ -4,7 +4,7 @@ Keystone token providers
Tokens are used to interact with the various OpenStack APIs. The token type
issued by keystone is configurable through the :file:`etc/keystone.conf` file.
Currently, there are four supported token types and they include UUID, Fernet,
Currently, there are four supported token types and they include UUID, fernet,
PKI, and PKIZ.
UUID tokens
@ -18,13 +18,14 @@ validate it.
Fernet tokens
~~~~~~~~~~~~~
Fernet tokens were introduced in the OpenStack Kilo release. Unlike the other
token types mentioned in this document, Fernet tokens do not need to be
persisted in a back end. ``AES256`` encryption is used to protect the
The fernet token format was introduced in the OpenStack Kilo release. Unlike
the other token types mentioned in this document, fernet tokens do not need to
be persisted in a back end. ``AES256`` encryption is used to protect the
information stored in the token and integrity is verified with a ``SHA256
HMAC`` signature. Only the Identity service should have access to the keys used
to encrypt and decrypt Fernet tokens. Like UUID tokens, Fernet tokens must be
passed back to the Identity service in order to validate them.
to encrypt and decrypt fernet tokens. Like UUID tokens, fernet tokens must be
passed back to the Identity service in order to validate them. For more
information on the fernet token type, see the :ref:`keystone_fernet_token_faq`.
PKI and PKIZ tokens
~~~~~~~~~~~~~~~~~~~