openstack/openstack-manuals
Code Issues Proposed changes
d32e63e8ad
openstack-manuals/doc/common/section_identity-troubleshooting.xml
Bruce Reeler 29e7164124 Run keystone-manage token_flush cmd 
Because the Identity server's token database table grows unconditionally
over time as new tokens are generated, the token table needs to be
manually cleared. The administrator must run the keystone-manage token_flush
command on a regular basis. Recommended frreqencyis once every minute.

Change-Id: Ie3dbe5e4336af71af7b83d68d28a1a72471f53eb
Closes-Bug: #1315236
2014-05-02 12:22:25 -05:00

185 lines
10 KiB
XML
Raw Blame History

<?xml version="1.0" encoding="UTF-8"?>
<section xml:id="troubleshooting-identity-service"
xmlns="http://docbook.org/ns/docbook"
xmlns:xi="http://www.w3.org/2001/XInclude"
xmlns:xlink="http://www.w3.org/1999/xlink" version="5.0">
<title>Troubleshoot the Identity service</title>
<para>To troubleshoot the Identity service, review the logs in the
<filename>/var/log/keystone.log</filename> file.</para>
<note>
<para>Use the <filename>/etc/keystone/logging.conf</filename>
file to configure the location of log files.</para>
</note>
<para>The logs show the components that have come in to the WSGI
request, and ideally show an error that explains why an
authorization request failed. If you do not see the request in
the logs, run keystone with <parameter>--debug</parameter>
parameter. Pass the <parameter>--debug</parameter> parameter
before the command parameters.</para>
<section xml:id="debugging-pki-middleware">
<title>Debug PKI middleware</title>
<para>If you receive an <literal>Invalid OpenStack Identity
Credentials</literal> message when you talk to an
OpenStack service, it might be caused by the changeover
from UUID tokens to PKI tokens in the Grizzly release.
Learn how to troubleshoot this error.</para>
<para>The PKI-based token validation scheme relies on
certificates from Identity that are fetched through HTTP
and stored in a local directory. The location for this
directory is specified by the
<literal>signing_dir</literal> configuration option.
In your services configuration file, look for a section
like this:</para>
<programlisting language="ini">[keystone_authtoken]
signing_dir = /var/cache/glance/api
auth_uri = http://127.0.0.1:5000/
auth_host = 127.0.0.1
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = glance</programlisting>
<para>If your service lacks this stanza, the <link
xlink:href="https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/middleware/auth_token.py#L198"
>keystoneclient/middleware/auth_token.py</link> file
specifies the defaults. If no value is specified for this
directory, it <link
xlink:href="https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/middleware/auth_token.py#L299"
>defaults to a secure temporary directory.</link>
Initialization code for the service checks that the
directory exists and is writable. If it does not exist,
the code tries to create it. If this fails, the service
fails to start. However, it often succeeds but problems
occur later.</para>
<para>The first thing to check is that the
<literal>signing_dir</literal> does, in fact, exist.
If it does, check for the presence of the certificate
files inside there:</para>
<screen><prompt>$</prompt> <userinput>ls -la /var/cache/glance/api/</userinput></screen>
<screen><computeroutput>total 24
drwx------. 2 ayoung root 4096 Jul 22 10:58 .
drwxr-xr-x. 4 root root 4096 Nov 7 2012 ..
-rw-r-----. 1 ayoung ayoung 1424 Jul 22 10:58 cacert.pem
-rw-r-----. 1 ayoung ayoung 15 Jul 22 10:58 revoked.pem
-rw-r-----. 1 ayoung ayoung 4518 Jul 22 10:58 signing_cert.pem</computeroutput></screen>
<para>This directory contains two certificates and the token
revocation list. If these files are not present, your
service cannot fetch them from Identity. To troubleshoot,
try to talk to Identity to make sure it correctly serves
files, as follows:</para>
<screen><prompt>$</prompt> <userinput>curl http://localhost:35357/v2.0/certificates/signing</userinput></screen>
<para>This command fetches the signing certificate:</para>
<programlisting>Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: sha1WithRSAEncryption
Issuer: C=US, ST=Unset, L=Unset, O=Unset, CN=www.example.com
Validity
Not Before: Jul 22 14:57:31 2013 GMT
Not After : Jul 20 14:57:31 2023 GMT
Subject: C=US, ST=Unset, O=Unset, CN=www.example.com</programlisting>
<para>Note the expiration dates of the certificate:</para>
<programlisting>Not Before: Jul 22 14:57:31 2013 GMT
Not After : Jul 20 14:57:31 2023 GMT</programlisting>
<para>The token revocation list is updated once a minute, but
the certificates are not. One possible problem is that the
certificates are the wrong files or garbage. You can
remove these files and run another command against your
server: They are fetched on demand.</para>
<para>The Identity service log should show the access of the
certificate files. You might have to turn up your logging
levels. Set <literal>debug = True</literal> and
<literal>verbose = True</literal> in your Identity
configuration file and restart the Identity server.</para>
<programlisting>(keystone.common.wsgi): 2013-07-24 12:18:11,461 DEBUG wsgi __call__
arg_dict: {}
(access): 2013-07-24 12:18:11,462 INFO core __call__ 127.0.0.1 - - [24/Jul/2013:16:18:11 +0000]
"GET http://localhost:35357/v2.0/certificates/signing HTTP/1.0" 200 4518</programlisting>
<para>If the files do not appear in your directory after this,
it is likely one of the following issues:</para>
<itemizedlist>
<listitem>
<para>Your service is configured incorrectly and
cannot talk to Identity. Check the
<literal>auth_port</literal> and
<literal>auth_host</literal> values and make
sure that you can talk to that service through
cURL, as shown previously.</para>
</listitem>
<listitem>
<para>Your signing directory is not writable. Use the
<command>chmod</command> command to change its
permissions so that the service (POSIX) user can
write to it. Verify the change through
<command>su</command> and
<command>touch</command> commands.</para>
</listitem>
<listitem>
<para>The SELinux policy is denying access to the
directory.</para>
</listitem>
</itemizedlist>
<para>SELinux troubles often occur when you use
Fedora/RHEL-based packages and you choose configuration
options that do not match the standard policy. Run the
<command>setenforce permissive</command> command. If
that makes a difference, you should relabel the directory.
If you are using a sub-directory of the
<filename>/var/cache/</filename> directory, run the
following command:</para>
<screen><prompt>#</prompt> <userinput>restorecon /var/cache/</userinput></screen>
<para>If you are not using a <filename>/var/cache</filename>
sub-directory, you should. Modify the
<literal>signing_dir</literal> configuration option
for your service and restart.</para>
<para>Set back to <literal>setenforce enforcing</literal> to
confirm that your changes solve the problem.</para>
<para>If your certificates are fetched on demand, the PKI
validation is working properly. Most likely, the token
from Identity is not valid for the operation you are
attempting to perform, and your user needs a different
role for the operation.</para>
</section>
<section xml:id="debugging-signing-key-file">
<title>Debug signing key file errors</title>
<para>If an error occurs when the signing key file opens, it
is possible that the person who ran the
<command>keystone-manage pki_setup</command> command
to generate certificates and keys did not use the correct
user. When you run the <command>keystone-manage
pki_setup</command> command, Identity generates a set
of certificates and keys in
<filename>/etc/keystone/ssl*</filename>, which is
owned by root:root.</para>
<para>This can present a problem when you run the Identity
daemon under the keystone user account (nologin) when you
try to run PKI. Unless you run the
<command>chown</command> command against the files
keystone:keystone or run the <command>keystone-manage
pki_setup</command> command with the
<parameter>--keystone-user</parameter> and
<parameter>--keystone-group</parameter> parameters,
you get an error, as follows:</para>
<screen><?db-font-size 75%?><computeroutput>2012-07-31 11:10:53 ERROR [keystone.common.cms] Error opening signing key file
/etc/keystone/ssl/private/signing_key.pem
140380567730016:error:0200100D:system library:fopen:Permission
denied:bss_file.c:398:fopen('/etc/keystone/ssl/private/signing_key.pem','r')
140380567730016:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:400:
unable to load signing key file</computeroutput></screen>
</section>
<section
xml:id="flushing-expired-tokens-from-token-database-table">
<title>Flush expired tokens from the token database
table</title>
<para>As you generate tokens, the token database table on the
Identity server grows. To clear the token table, an
administrative user must run the <command>keystone-manage
token_flush</command> command to flush the tokens.
When you flush tokens, expired tokens are deleted and
traceability is eliminated.</para>
<para>Use <command>cron</command> to schedule this command to
run frequently based on your workload. For large
workloads, running it every minute is recommended.</para>
</section>
</section>
View Git Blame Copy Permalink