To troubleshoot the Identity Service, review the logs in the /var/log/keystone.log file. Use the /etc/keystone/logging.conf file to configure the location of log files. The logs show the components that have come in to the WSGI request, and ideally show an error that explains why an authorization request failed. If you do not see the request in the logs, run keystone with --debug parameter. Pass the --debug parameter before the command parameters.

If you receive an Invalid Openstack Identity Credentials message when you talk to an OpenStack service, it might be caused by the changeover from UUID tokens to PKI tokens in the Grizzly release. Learn how to troubleshoot this error. The PKI-based token validation scheme relies on certificates from the Identity Service that are fetched through HTTP and stored in a local directory. The location for this directory is specified by the signing_dir configuration option. In your services configuration file, look for a section like this: [keystone_authtoken] signing_dir = /var/cache/glance/api auth_uri = http://127.0.0.1:5000/ auth_host = 127.0.0.1 auth_port = 35357 auth_protocol = http admin_tenant_name = service admin_user = glance If your service lacks this stanza, the keystoneclient/middleware/auth_token.py file specifies the defaults. If no value is specified for this directory, it defaults to a secure temporary directory. Initialization code for the service checks that the directory exists and is writable. If it does not exist, the code tries to create it. If this fails, the service fails to start. However, it often succeeds but problems occur later. The first thing to check is that the signing_dir does, in fact, exist. If it does, check for the presence of the certificate files inside there: $ ls -la /var/cache/glance/api/ total 24 drwx------. 2 ayoung root 4096 Jul 22 10:58 . drwxr-xr-x. 4 root root 4096 Nov 7 2012 .. -rw-r-----. 1 ayoung ayoung 1424 Jul 22 10:58 cacert.pem -rw-r-----. 1 ayoung ayoung 15 Jul 22 10:58 revoked.pem -rw-r-----. 1 ayoung ayoung 4518 Jul 22 10:58 signing_cert.pem This directory contains two certificates and the token revocation list. If these files are not present, your service cannot fetch them from the Identity Service. To troubleshoot, try to talk to the Identity Service to make sure it is serving out the files, as follows: $ curl http://localhost:35357/v2.0/certificates/signing This command fetches the signing certificate: Certificate: Data: Version: 3 (0x2) Serial Number: 1 (0x1) Signature Algorithm: sha1WithRSAEncryption Issuer: C=US, ST=Unset, L=Unset, O=Unset, CN=www.example.com Validity Not Before: Jul 22 14:57:31 2013 GMT Not After : Jul 20 14:57:31 2023 GMT Subject: C=US, ST=Unset, O=Unset, CN=www.example.com Note the expiration dates of the certificate: Not Before: Jul 22 14:57:31 2013 GMT Not After : Jul 20 14:57:31 2023 GMT The token revocation list is updated once a minute, but the certificates are not. One possible problem is that the certificates are the wrong files or garbage. You can remove these files and run another command against your server: They are fetched on demand. The Identity Service log should show the access of the certificate files. You might have to turn up your logging levels. Set debug = True and verbose = True in your Identity Service configuration file and restart the Identity Service server. (keystone.common.wsgi): 2013-07-24 12:18:11,461 DEBUG wsgi __call__ arg_dict: {} (access): 2013-07-24 12:18:11,462 INFO core __call__ 127.0.0.1 - - [24/Jul/2013:16:18:11 +0000] "GET http://localhost:35357/v2.0/certificates/signing HTTP/1.0" 200 4518 If the files do not appear in your directory after this, it is likely one of the following issues: Your service is configured incorrectly and cannot talk to the Identity Service. Check the auth_port and auth_host values and make sure that you can talk to that service through cURL, as shown previously. Your signing directory is not writable. Use the chmod command to change its permissions so that the service (POSIX) user can write to it. Verify the change through su and touch commands. The SELinux policy is denying access to the directory. SELinux troubles often when you use Fedora/RHEL-based packages and you choose configuration options that do not match the standard policy. Run the setenforce permissive command. If that makes a different, you should relabel the directory. If you are using a sub-directory of the /var/cache/ directory, run the following command: $ sudo restorecon /var/cache/ If you are not using a /var/cache sub-directory, you should. Modify the signing_dir configuration option for your service and restart. Set back to setenforce enforcing to confirm that your changes solve the problem. If your certificates are fetched on demand, the PKI validation is working properly. Most likely, the token from the Identity Service is not valid for the operation you are attempting to perform, and your user needs a different role for the operation.

If an error occurs when the signing key file opens, it is possible that the person who ran the keystone-manage pki_setup command to generate certificates and keys did not use the correct user. When you run the keystone-manage pki_setup command, the Identity Service generates a set of certificates and keys in /etc/keystone/ssl*, which is owned by root:root. This can present a problem when you run the Identity Service daemon under the keystone user account (nologin) when you try to run PKI. Unless you run the chown command against the files keystone:keystone or run the keystone-manage pki_setup command with the --keystone-user and --keystone-group parameters, you get an error, as follows: 2012-07-31 11:10:53 ERROR [keystone.common.cms] Error opening signing key file /etc/keystone/ssl/private/signing_key.pem 140380567730016:error:0200100D:system library:fopen:Permission denied:bss_file.c:398:fopen('/etc/keystone/ssl/private/signing_key.pem','r') 140380567730016:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:400: unable to load signing key file