Add Fernet FAQ

We've been getting various questions on Fernet tokens for a while. We should
add some of the common ones to docs.

Change-Id: I8dd42ef1657cf6312636f1f59a5897b6956a4ee4

doc/admin-guide-cloud/source/identity_management.rst

@@ -21,6 +21,7 @@ command-line client.
    keystone_integrate_with_ldap.rst
    keystone_tokens.rst
    keystone_token-binding.rst
+   keystone_fernet_token_faq.rst
    keystone_use_trusts.rst
    keystone_caching_layer.rst
    identity_user_crud.rst

doc/admin-guide-cloud/source/keystone_fernet_token_faq.rst

@@ -0,0 +1,325 @@
+===================================
+fernet - Frequently Asked Questions
+===================================
+
+The following questions have been asked periodically since the initial release
+of the fernet token format in Kilo.
+
+What are the different types of keys?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A key repository is required by keystone in order to create fernet tokens.
+These keys are used to encrypt and decrypt the information that makes up the
+payload of the token. Each key in the repository can have one of three states.
+The state of the key determines how keystone uses a key with fernet tokens. The
+different types are as follows:
+
+Primary key:
+  There is only ever one primary key in a key repository. The primary key is
+  allowed to encrypt and decrypt tokens. This key is always named as the
+  highest index in the repository.
+Secondary key:
+  A secondary key was at one point a primary key, but has been demoted in place
+  of another primary key. It is only allowed to decrypt tokens. Since it was
+  the primary at some point in time, its existence in the key repository is
+  justified. Keystone needs to be able to decrypt tokens that were created with
+  old primary keys.
+Staged key:
+  The staged key is a special key that shares some similarities with secondary
+  keys. There can only ever be one staged key in a repository and it must
+  exist. Just like secondary keys, staged keys have the ability to decrypt
+  tokens. Unlike secondary keys, staged keys have never been a primary key. In
+  fact, they are opposites since the staged key will always be the next primary
+  key. This helps clarify the name because they are the next key staged to be
+  the primary key. This key is always named as ``0`` in the key repository.
+
+So, how does a staged key help me and why do I care about it?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The fernet keys have a natural lifecycle. Each key starts as a staged key, is
+promoted to be the primary key, and then demoted to be a secondary key. New
+tokens can only be encrypted with a primary key. Secondary and staged keys are
+never used to encrypt token. The staged key is a special key given the order of
+events and the attributes of each type of key. The staged key is the only key
+in the repository that has not had a chance to encrypt any tokens yet, but it
+is still allowed to decrypt tokens. As an operator, this gives you the chance
+to perform a key rotation on one keystone node, and distribute the new key set
+over a span of time. This does not require the distribution to take place in an
+ultra short period of time. Tokens encrypted with a primary key can be
+decrypted, and validated, on other nodes where that key is still staged.
+
+Where do I put my key repository?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The key repository is specified using the ``key_repository`` option in the
+keystone configuration file. The keystone process should be able to read and
+write to this location but it should be kept secret otherwise. Currently,
+keystone only supports file-backed key repositories.
+
+..code-block:: ini
+
+  [fernet_tokens]
+  key_repository = /etc/keystone/fernet-keys/
+
+What is the recommended way to rotate and distribute keys?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``keystone-manage`` command line utility includes a key rotation mechanism.
+This mechanism will initialize and rotate keys but does not make an effort to
+distribute keys across keystone nodes. The distribution of keys across a
+keystone deployment is best handled through configuration management tooling.
+Use ``keystone-manage fernet_rotate`` to rotate the key repository.
+
+Do fernet tokens still expires?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Yes, fernet tokens can expire just like any other keystone token formats.
+
+Why should I choose fernet tokens over UUID tokens?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Even though fernet tokens operate very similarly to UUID tokens, they do not
+require persistence. The keystone token database no longer suffers bloat as a
+side effect of authentication. Pruning expired tokens from the token database
+is no longer required when using fernet tokens. Because fernet tokens do not
+require persistence, they do not have to be replicated. As long as each
+keystone node shares the same key repository, fernet tokens can be created and
+validated instantly across nodes.
+
+Why should I choose fernet tokens over PKI or PKIZ tokens?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The arguments for using fernet over PKI and PKIZ remain the same as UUID, in
+addition to the fact that fernet tokens are much smaller than PKI and PKIZ
+tokens. PKI and PKIZ tokens still require persistent storage and can sometimes
+cause issues due to their size. This issue is mitigated when switching to
+fernet because fernet tokens are kept under a 250 byte limit. PKI and PKIZ
+tokens typically exceed 1600 bytes in length. The length of a PKI or PKIZ token
+is dependent on the size of the deployment. Bigger service catalogs will result
+in longer token lengths. This pattern does not exist with fernet tokens because
+the contents of the encrypted payload is kept to a minimum.
+
+Should I rotate and distribute keys from the same keystone node every rotation?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+No, but the relationship between rotation and distribution should be lock-step.
+Once you rotate keys on one keystone node, the key repository from that node
+should be distributed to the rest of the cluster. Once you confirm that each
+node has the same key repository state, you could rotate and distribute from
+any other node in the cluster.
+
+If the rotation and distribution are not lock-step, a single keystone node in
+the deployment will create tokens with a primary key that no other node has as
+a staged key. This will cause tokens generated from one keystone node to fail
+validation on other keystone nodes.
+
+How do I add new keystone nodes to a deployment?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The keys used to create fernet tokens should be treated like super secret
+configuration files, similar to an SSL secret key. Before a node is allowed to
+join an existing cluster, issuing and validating tokens, it should have the
+same key repository as the rest of the nodes in the cluster.
+
+How should I approach key distribution?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Remember that key distribution is only required in multi-node keystone
+deployments. If you only have one keystone node serving requests in your
+deployment, key distribution is unnecessary.
+
+Key distribution is a problem best approached from the deployment's current
+configuration management system. Since not all deployments use the same
+configuration management systems, it makes sense to explore options around what
+is already available for managing keys, while keeping the secrecy of the keys
+in mind. Many configuration management tools can leverage something like
+``rsync`` to manage key distribution.
+
+Key rotation is a single operation that promotes the current staged key to
+primary, creates a new staged key, and prunes old secondary keys. It is easiest
+to do this on a single node and verify the rotation took place properly before
+distributing the key repository to the rest of the cluster. The concept behind
+the staged key breaks the expectation that key rotation and key distribution
+have to be done in a single step. With the staged key, we have time to inspect
+the new key repository before syncing state with the rest of the cluster. Key
+distribution should be an operation that can run in succession until it
+succeeds. The following might help illustrate the isolation between key
+rotation and key distribution.
+
+1. Ensure all keystone nodes in the deployment have the same key repository.
+2. Pick a keystone node in the cluster to rotate from.
+3. Rotate keys.
+    a) Was is successful?
+        i) If no, investigate issues with the particular keystone node you
+           rotated keys on. Fernet keys are small and the operation for
+           rotation is trivial. There should not be much room for error in key
+           rotation. It is possible that the user does not have the ability to
+           write new keys to the key repository. Log output from
+           ``keystone-manage fernet_rotate`` should give more information into
+           specific failures.
+        ii) If yes, you should see a new staged key. The old staged key should
+            be the new primary. Depending on the ``max_active_keys`` limit you
+            might have secondary keys that were pruned. At this point, the node
+            that you rotated on will be creating fernet tokens with a primary
+            key that all other nodes should have as the staged key. This is why
+            we checked the state of all key repositories in Step one. All other
+            nodes in the cluster should be able to decrypt tokens created with
+            the new primary key. At this point, we are ready to distribute the
+            new key set.
+4. Distribute the new key repository.
+    a) Was it successful?
+        i) If yes, you should be able to confirm that all nodes in the cluster
+           have the same key repository that was introduced in Step 3.  All
+           nodes in the cluster will be creating tokens with the primary key
+           that was promoted in Step 3. No further action is required until the
+           next schedule key rotation.
+        ii) If no, try distributing again. Remember that we already rotated the
+            repository and performing another rotation at this point will
+            result in tokens that cannot be validated across certain hosts.
+            Specifically, the hosts that did not get the latest key set. You
+            should be able to distribe keys until it is successful. If certain
+            nodes have issues syncing, it could be permission or network issues
+            and those should be resolved before subsequent rotations.
+
+How long should I keep my keys around?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The fernet tokens that keystone creates are only secure as the keys creating
+them. With staged keys the penalty of key rotation is low, allowing you to err
+on the side of security and rotate weekly, daily, or even hourly.  Ultimately,
+this should be less time than it takes an attacker to break a ``AES256`` key
+and a ``SHA256 HMAC``.
+
+Is a fernet token still a bearer token?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Yes, and they follow exactly the same validation path as UUID tokens, with the
+exception of being written to, and read from, a back end. If someone
+compromises your fernet token, they have the power to do all the operations you
+are allowed to do.
+
+What if I need to revoke all my tokens?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To invalidate every token issued from keystone and start fresh, remove the
+current key repository, create a new key set, and redistribute it to all nodes
+in the cluster. This will render every token issued from keystone as invalid
+regardless if the token has actually expired. When a client goes to
+re-authenticate, the new token will have been created with a new fernet key.
+
+What can an attacker do if they compromise a fernet key in my deployment?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If any key used in the key repository is compromised, an attacker will be able
+to build their own tokens. If they know the ID of an administrator on a
+project, they could generate administrator tokens for the project. They will be
+able to generate their own tokens until the compromised key has been removed
+from from the repository.
+
+I rotated keys and now tokens are invalidating early, what did I do?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Using fernet tokens requires some awareness around token expiration and the key
+lifecycle. You do not want to rotate so often that secondary keys are removed
+that might still be needed to decrypt unexpired tokens. If this happens, you
+will not be able to decrypt the token because the key the was used to encrypt
+it is now gone. Only remove keys that you know are not being used to encrypt or
+decrypt tokens.
+
+For example, your token is valid for 24 hours and we want to rotate keys every
+six hours. We will need to make sure tokens that were created at 08:00 AM on
+Monday are still valid at 07:00 AM on Tuesday, assuming they were not
+prematurely revoked. To accomplish this, we will want to make sure we set
+``max_active_keys=6`` in our keystone configuration file. This will allow us to
+hold all keys that might still be required to validate a previous token, but
+keeps the key repository limited to only the keys that are needed.
+
+The number of ``max_active_keys`` for a deployment can be determined by
+dividing the token lifetime, in hours, by the frequency of rotation in hours
+and adding two. Better illustrated as::
+
+    token_expiration = 24
+    rotation_frequency = 6
+    max_active_keys = (token_expiration / rotation_frequency) + 2
+
+The reason for adding two additional keys to the count is to include the staged
+key and a buffer key. This can be shown based on the previous example. We
+initially setup the key repository at 6:00 AM on Monday, and the initial state
+looks like::
+
+    $ ls -la /etc/keystone/fernet-keys/
+    drwx------ 2 keystone keystone 4096 .
+    drwxr-xr-x 3 keystone keystone 4096 ..
+    -rw------- 1 keystone keystone   44 0    (staged key)
+    -rw------- 1 keystone keystone   44 1    (primary key)
+
+All tokens created after 6:00 AM are encrypted with key ``1``. At 12:00 PM we
+will rotate keys again, resulting in::
+
+    $ ls -la /etc/keystone/fernet-keys/
+    drwx------ 2 keystone keystone 4096 .
+    drwxr-xr-x 3 keystone keystone 4096 ..
+    -rw------- 1 keystone keystone   44 0    (staged key)
+    -rw------- 1 keystone keystone   44 1    (secondary key)
+    -rw------- 1 keystone keystone   44 2    (primary key)
+
+We are still able to validate tokens created between 6:00 - 11:59 AM because
+the ``1`` key still exists as a secondary key. All tokens issued after 12:00 PM
+will be encrypted with key ``2``. At 6:00 PM we do our next rotation, resulting
+in::
+
+    $ ls -la /etc/keystone/fernet-keys/
+    drwx------ 2 keystone keystone 4096 .
+    drwxr-xr-x 3 keystone keystone 4096 ..
+    -rw------- 1 keystone keystone   44 0    (staged key)
+    -rw------- 1 keystone keystone   44 1    (secondary key)
+    -rw------- 1 keystone keystone   44 2    (secondary key)
+    -rw------- 1 keystone keystone   44 3    (primary key)
+
+It is still possible to validate tokens issued from 6:00 AM - 5:59 PM because
+keys ``1`` and ``2`` exist as secondary keys. Every token issued until 11:59 PM
+will be encrypted with key ``3``, and at 12:00 AM we do our next rotation::
+
+    $ ls -la /etc/keystone/fernet-keys/
+    drwx------ 2 keystone keystone 4096 .
+    drwxr-xr-x 3 keystone keystone 4096 ..
+    -rw------- 1 keystone keystone   44 0    (staged key)
+    -rw------- 1 keystone keystone   44 1    (secondary key)
+    -rw------- 1 keystone keystone   44 2    (secondary key)
+    -rw------- 1 keystone keystone   44 3    (secondary key)
+    -rw------- 1 keystone keystone   44 4    (primary key)
+
+Just like before, we can still validate tokens issued from 6:00 AM the previous
+day until 5:59 AM today because keys ``1`` - ``4`` are present. At 6:00 AM,
+tokens issued from the previous day will start to expire and we do our next
+scheduled rotation::
+
+    $ ls -la /etc/keystone/fernet-keys/
+    drwx------ 2 keystone keystone 4096 .
+    drwxr-xr-x 3 keystone keystone 4096 ..
+    -rw------- 1 keystone keystone   44 0    (staged key)
+    -rw------- 1 keystone keystone   44 1    (secondary key)
+    -rw------- 1 keystone keystone   44 2    (secondary key)
+    -rw------- 1 keystone keystone   44 3    (secondary key)
+    -rw------- 1 keystone keystone   44 4    (secondary key)
+    -rw------- 1 keystone keystone   44 5    (primary key)
+
+Tokens will naturally expire after 6:00 AM, but we will not be able to remove
+key ``1`` until the next rotation because it encrypted all tokens from 6:00 AM
+to 12:00 PM the day before. Once we do our next rotation, which is at 12:00 PM,
+the ``1`` key will be pruned from the repository::
+
+    $ ls -la /etc/keystone/fernet-keys/
+    drwx------ 2 keystone keystone 4096 .
+    drwxr-xr-x 3 keystone keystone 4096 ..
+    -rw------- 1 keystone keystone   44 0    (staged key)
+    -rw------- 1 keystone keystone   44 2    (secondary key)
+    -rw------- 1 keystone keystone   44 3    (secondary key)
+    -rw------- 1 keystone keystone   44 4    (secondary key)
+    -rw------- 1 keystone keystone   44 5    (secondary key)
+    -rw------- 1 keystone keystone   44 6    (primary key)
+
+If keystone were to receive a token that was created between 6:00 AM and 12:00
+PM the day before, encrypted with the ``1`` key, it would not be valid because
+it was already expired. This makes it possible for us to remove the ``1`` key
+from the repository without negative validation side-effects.

doc/admin-guide-cloud/source/keystone_tokens.rst

@@ -4,7 +4,7 @@ Keystone token providers
 
 Tokens are used to interact with the various OpenStack APIs. The token type
 issued by keystone is configurable through the :file:`etc/keystone.conf` file.
-Currently, there are four supported token types and they include UUID, Fernet,
+Currently, there are four supported token types and they include UUID, fernet,
 PKI, and PKIZ.
 
 UUID tokens
@@ -18,13 +18,14 @@ validate it.
 Fernet tokens
 ~~~~~~~~~~~~~
 
-Fernet tokens were introduced in the OpenStack Kilo release. Unlike the other
+The fernet token format was introduced in the OpenStack Kilo release. Unlike
-token types mentioned in this document, Fernet tokens do not need to be
+the other token types mentioned in this document, fernet tokens do not need to
-persisted in a back end. ``AES256`` encryption is used to protect the
+be persisted in a back end. ``AES256`` encryption is used to protect the
 information stored in the token and integrity is verified with a ``SHA256
 HMAC`` signature. Only the Identity service should have access to the keys used
-to encrypt and decrypt Fernet tokens. Like UUID tokens, Fernet tokens must be
+to encrypt and decrypt fernet tokens. Like UUID tokens, fernet tokens must be
-passed back to the Identity service in order to validate them.
+passed back to the Identity service in order to validate them. For more
+information on the fernet token type, see the :ref:`keystone_fernet_token_faq`.
 
 PKI and PKIZ tokens
 ~~~~~~~~~~~~~~~~~~~