openstack/openstack-manuals
Code Issues Proposed changes

Remove ha-guide docs to avoid conflict

Browse Source 
The ha-guide repo is being restored and owned by the Operations
Documentation SIG. Until the openstack-manuals content is removed,
patches merged to either repo will overwrite the content of the web
site.

Depends-on: https://review.openstack.org/#/c/603219/

Change-Id: Iabd74139d1d73016563d0cfd30cf8ed6a3e1fc57
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
This commit is contained in:
Sean McGinnis 2018-09-17 21:25:33 -05:00
parent eb94121385
commit 67bad97c52
No known key found for this signature in database
GPG Key ID: CE7EE4BFAF8D70C8
52 changed files with 11 additions and 16127 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
doc-tools-check-languages.conf
View File

@ -7,10 +7,10 @@ declare -A BOOKS=(
["de"]="image-guide install-guide"
["fr"]="install-guide"
["id"]="image-guide install-guide"
["ja"]="ha-guide image-guide install-guide"
["ja"]="image-guide install-guide"
["ko_KR"]="install-guide"
["ru"]="install-guide"
["tr_TR"]="arch-design ha-guide image-guide install-guide"
["tr_TR"]="arch-design image-guide install-guide"
["zh_CN"]="install-guide"
)
@ -24,7 +24,6 @@ declare -A SPECIAL_BOOKS=(
# This needs special handling, handle it with the RST tools.
["common"]="RST"
["glossary"]="RST"
["ha-guide"]="RST"
["image-guide"]="RST"
["install-guide"]="RST"
# Do not translate

2
doc/arch-design/source/design-storage/design-storage-arch.rst
View File

@ -496,7 +496,7 @@ servers and load balancing. HAProxy is one method of providing load
balancing and high availability and is often combined with keepalived
or pacemaker to ensure the HAProxy service maintains a stable VIP.
Sample HAProxy configurations can be found in the `OpenStack HA Guide.
<https://docs.openstack.org/ha-guide/controller-ha-haproxy.html#configuring-haproxy>`_.
<https://docs.openstack.org/ha-guide/>`_.
Replication
-----------

3
doc/doc-contrib-guide/source/doc-bugs.rst
View File

@ -249,8 +249,7 @@ Depending on the area a bug affects, it has one or more tags. For example:
* **low-hanging-fruit** for documentation bugs that are straightforward to fix.
If you are a newcomer, this is a way to start.
* **ha-guide**, **install-guide**, **image-guide**, and other for specific
guides.
* **install-guide**, **image-guide**, and other for specific guides.
* **infra**, **theme** for documentation bugs that are in the documentation
build tool chain.

3
doc/doc-contrib-guide/source/topic-tags.rst
View File

@ -18,9 +18,6 @@ to, use the following tags:
[doc-contrib]
OpenStack Documentation Contributor Guide
[ha-guide]
OpenStack High Availability Guide
[image-guide]
OpenStack Virtual Machine Image Guide

27
doc/ha-guide/setup.cfg
View File

@ -1,27 +0,0 @@
[metadata]
name = openstackhaguide
summary = OpenStack High Availability Guide
author = OpenStack
author-email = openstack-dev@lists.openstack.org
home-page = https://docs.openstack.org/
classifier =
Environment :: OpenStack
Intended Audience :: Information Technology
Intended Audience :: System Administrators
License :: OSI Approved :: Apache Software License
Operating System :: POSIX :: Linux
Topic :: Documentation
[global]
setup-hooks =
pbr.hooks.setup_hook
[files]
[build_sphinx]
warning-is-error = 1
build-dir = build
source-dir = source
[wheel]
universal = 1

30
doc/ha-guide/setup.py
View File

@ -1,30 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# THIS FILE IS MANAGED BY THE GLOBAL REQUIREMENTS REPO - DO NOT EDIT
import setuptools
# In python < 2.7.4, a lazy loading of package `pbr` will break
# setuptools if some other modules registered functions in `atexit`.
# solution from: http://bugs.python.org/issue15881#msg170215
try:
import multiprocessing # noqa
except ImportError:
pass
setuptools.setup(
setup_requires=['pbr'],
pbr=True)

8
doc/ha-guide/source/appendix.rst
View File

@ -1,8 +0,0 @@
Appendix
~~~~~~~~
.. toctree::
:maxdepth: 1
common/app-support.rst
common/glossary.rst

1
doc/ha-guide/source/common
View File

@ -1 +0,0 @@
../../common

52
doc/ha-guide/source/compute-node-ha.rst
View File

@ -1,52 +0,0 @@
============================
Configuring the compute node
============================
The `Installation Guides
<https://docs.openstack.org/ocata/install/>`_
provide instructions for installing multiple compute nodes.
To make the compute nodes highly available, you must configure the
environment to include multiple instances of the API and other services.
Configuring high availability for instances
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
As of September 2016, the OpenStack High Availability community is
designing and developing an official and unified way to provide high
availability for instances. We are developing automatic
recovery from failures of hardware or hypervisor-related software on
the compute node, or other failures that could prevent instances from
functioning correctly, such as, issues with a cinder volume I/O path.
More details are available in the `user story
<https://specs.openstack.org/openstack/openstack-user-stories/user-stories/proposed/ha_vm.html>`_
co-authored by OpenStack's HA community and `Product Working Group
<https://wiki.openstack.org/wiki/ProductTeam>`_ (PWG), where this feature is
identified as missing functionality in OpenStack, which
should be addressed with high priority.
Existing solutions
~~~~~~~~~~~~~~~~~~
The architectural challenges of instance HA and several currently
existing solutions were presented in `a talk at the Austin summit
<https://www.openstack.org/videos/video/high-availability-for-pets-and-hypervisors-state-of-the-nation>`_,
for which `slides are also available <http://aspiers.github.io/openstack-summit-2016-austin-compute-ha/>`_.
The code for three of these solutions can be found online at the following
links:
* `a mistral-based auto-recovery workflow
<https://github.com/gryf/mistral-evacuate>`_, by Intel
* `masakari <https://launchpad.net/masakari>`_, by NTT
* `OCF RAs
<https://aspiers.github.io/openstack-summit-2016-austin-compute-ha/#/ocf-pros-cons>`_,
as used by Red Hat and SUSE
Current upstream work
~~~~~~~~~~~~~~~~~~~~~
Work is in progress on a unified approach, which combines the best
aspects of existing upstream solutions. More details are available on
`the HA VMs user story wiki
<https://wiki.openstack.org/wiki/ProductTeam/User_Stories/HA_VMs>`_.

299
doc/ha-guide/source/conf.py
View File

@ -1,299 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
import os
# import sys
import openstackdocstheme
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
# sys.path.insert(0, os.path.abspath('.'))
# -- General configuration ------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = ['openstackdocstheme']
# Add any paths that contain templates here, relative to this directory.
# templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
# The encoding of source files.
# source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
repository_name = "openstack/openstack-manuals"
bug_project = 'openstack-manuals'
project = u'High Availability Guide'
bug_tag = u'ha-guide'
copyright = u'2015-2018, OpenStack contributors'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = ''
# The full version, including alpha/beta/rc tags.
release = ''
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
# language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
# today = ''
# Else, today_fmt is used as the format for a strftime call.
# today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = ['common/cli*', 'common/nova*',
'common/appendix.rst',
'common/get-started*', 'common/dashboard*']
# The reST default role (used for this markup: `text`) to use for all
# documents.
# default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
# add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
# add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
# show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
# modindex_common_prefix = []
# If true, keep warnings as "system message" paragraphs in the built documents.
# keep_warnings = False
# -- Options for HTML output ----------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'openstackdocs'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
html_theme_options = {
'display_badge': False
}
# Add any paths that contain custom themes here, relative to this directory.
# html_theme_path = [openstackdocstheme.get_html_theme_path()]
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
# html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
# html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
# html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
# html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
# html_static_path = []
# Add any extra paths that contain custom files (such as robots.txt or
# .htaccess) here, relative to this directory. These files are copied
# directly to the root of the documentation.
# html_extra_path = []
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
# So that we can enable "log-a-bug" links from each output HTML page, this
# variable must be set to a format that includes year, month, day, hours and
# minutes.
html_last_updated_fmt = '%Y-%m-%d %H:%M'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
# html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
# html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
# html_additional_pages = {}
# If false, no module index is generated.
# html_domain_indices = True
# If false, no index is generated.
html_use_index = False
# If true, the index is split into individual pages for each letter.
# html_split_index = False
# If true, links to the reST sources are added to the pages.
html_show_sourcelink = False
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
# html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
# html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
# html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
# html_file_suffix = None
# Output file base name for HTML help builder.
htmlhelp_basename = 'ha-guide'
# If true, publish source files
html_copy_source = False
# -- Options for LaTeX output ---------------------------------------------
pdf_theme_path = openstackdocstheme.get_pdf_theme_path()
openstack_logo = openstackdocstheme.get_openstack_logo_path()
latex_custom_template = r"""
\newcommand{\openstacklogo}{%s}
\usepackage{%s}
""" % (openstack_logo, pdf_theme_path)
latex_engine = 'xelatex'
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
'papersize': 'a4paper',
# The font size ('10pt', '11pt' or '12pt').
'pointsize': '11pt',
#Default figure align
'figure_align': 'H',
# Not to generate blank page after chapter
'classoptions': ',openany',
# Additional stuff for the LaTeX preamble.
'preamble': latex_custom_template,
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
('index', 'HAGuide.tex', u'High Availability Guide',
u'OpenStack contributors', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
# latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
# latex_use_parts = False
# If true, show page references after internal links.
# latex_show_pagerefs = False
# If true, show URL addresses after external links.
# latex_show_urls = False
# Documents to append as an appendix to all manuals.
# latex_appendices = []
# If false, no module index is generated.
# latex_domain_indices = True
# -- Options for manual page output ---------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
('index', 'haguide', u'High Availability Guide',
[u'OpenStack contributors'], 1)
]
# If true, show URL addresses after external links.
# man_show_urls = False
# -- Options for Texinfo output -------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
('index', 'HAGuide', u'High Availability Guide',
u'OpenStack contributors', 'HAGuide',
'This guide shows OpenStack operators and deployers how to configure'
'OpenStack Networking to be robust and fault-tolerant.', 'Miscellaneous'),
]
# Documents to append as an appendix to all manuals.
# texinfo_appendices = []
# If false, no module index is generated.
# texinfo_domain_indices = True
# How to display URL addresses: 'footnote', 'no', or 'inline'.
# texinfo_show_urls = 'footnote'
# If true, do not generate a @detailmenu in the "Top" node's menu.
# texinfo_no_detailmenu = False
# -- Options for Internationalization output ------------------------------
locale_dirs = ['locale/']

233
doc/ha-guide/source/controller-ha-haproxy.rst
View File

@ -1,233 +0,0 @@
=======
HAProxy
=======
HAProxy provides a fast and reliable HTTP reverse proxy and load balancer
for TCP or HTTP applications. It is particularly suited for web crawling
under very high loads while needing persistence or Layer 7 processing.
It realistically supports tens of thousands of connections with recent
hardware.
Each instance of HAProxy configures its front end to accept connections only
to the virtual IP (VIP) address. The HAProxy back end (termination
point) is a list of all the IP addresses of instances for load balancing.
.. note::
Ensure your HAProxy installation is not a single point of failure,
it is advisable to have multiple HAProxy instances running.
You can also ensure the availability by other means, using Keepalived
or Pacemaker.
Alternatively, you can use a commercial load balancer, which is hardware
or software. We recommend a hardware load balancer as it generally has
good performance.
For detailed instructions about installing HAProxy on your nodes,
see the HAProxy `official documentation <http://www.haproxy.org/#docs>`_.
Configuring HAProxy
~~~~~~~~~~~~~~~~~~~
#. Restart the HAProxy service.
#. Locate your HAProxy instance on each OpenStack controller node in your
environment. The following is an example ``/etc/haproxy/haproxy.cfg``
configuration file. Configure your instance using the following
configuration file, you will need a copy of it on each
controller node.
.. code-block:: none
global
chroot /var/lib/haproxy
daemon
group haproxy
maxconn 4000
pidfile /var/run/haproxy.pid
user haproxy
defaults
log global
maxconn 4000
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout check 10s
listen dashboard_cluster
bind <Virtual IP>:443
balance source
option tcpka
option httpchk
option tcplog
server controller1 10.0.0.12:443 check inter 2000 rise 2 fall 5
server controller2 10.0.0.13:443 check inter 2000 rise 2 fall 5
server controller3 10.0.0.14:443 check inter 2000 rise 2 fall 5
listen galera_cluster
bind <Virtual IP>:3306
balance source
option mysql-check
server controller1 10.0.0.12:3306 check port 9200 inter 2000 rise 2 fall 5
server controller2 10.0.0.13:3306 backup check port 9200 inter 2000 rise 2 fall 5
server controller3 10.0.0.14:3306 backup check port 9200 inter 2000 rise 2 fall 5
listen glance_api_cluster
bind <Virtual IP>:9292
balance source
option tcpka
option httpchk
option tcplog
server controller1 10.0.0.12:9292 check inter 2000 rise 2 fall 5
server controller2 10.0.0.13:9292 check inter 2000 rise 2 fall 5
server controller3 10.0.0.14:9292 check inter 2000 rise 2 fall 5
listen glance_registry_cluster
bind <Virtual IP>:9191
balance source
option tcpka
option tcplog
server controller1 10.0.0.12:9191 check inter 2000 rise 2 fall 5
server controller2 10.0.0.13:9191 check inter 2000 rise 2 fall 5
server controller3 10.0.0.14:9191 check inter 2000 rise 2 fall 5
listen keystone_admin_public_internal_cluster
bind <Virtual IP>:5000
balance source
option tcpka
option httpchk
option tcplog
server controller1 10.0.0.12:5000 check inter 2000 rise 2 fall 5
server controller2 10.0.0.13:5000 check inter 2000 rise 2 fall 5
server controller3 10.0.0.14:5000 check inter 2000 rise 2 fall 5
listen nova_ec2_api_cluster
bind <Virtual IP>:8773
balance source
option tcpka
option tcplog
server controller1 10.0.0.12:8773 check inter 2000 rise 2 fall 5
server controller2 10.0.0.13:8773 check inter 2000 rise 2 fall 5
server controller3 10.0.0.14:8773 check inter 2000 rise 2 fall 5
listen nova_compute_api_cluster
bind <Virtual IP>:8774
balance source
option tcpka
option httpchk
option tcplog
server controller1 10.0.0.12:8774 check inter 2000 rise 2 fall 5
server controller2 10.0.0.13:8774 check inter 2000 rise 2 fall 5
server controller3 10.0.0.14:8774 check inter 2000 rise 2 fall 5
listen nova_metadata_api_cluster
bind <Virtual IP>:8775
balance source
option tcpka
option tcplog
server controller1 10.0.0.12:8775 check inter 2000 rise 2 fall 5
server controller2 10.0.0.13:8775 check inter 2000 rise 2 fall 5
server controller3 10.0.0.14:8775 check inter 2000 rise 2 fall 5
listen cinder_api_cluster
bind <Virtual IP>:8776
balance source
option tcpka
option httpchk
option tcplog
server controller1 10.0.0.12:8776 check inter 2000 rise 2 fall 5
server controller2 10.0.0.13:8776 check inter 2000 rise 2 fall 5
server controller3 10.0.0.14:8776 check inter 2000 rise 2 fall 5
listen ceilometer_api_cluster
bind <Virtual IP>:8777
balance source
option tcpka
option tcplog
server controller1 10.0.0.12:8777 check inter 2000 rise 2 fall 5
server controller2 10.0.0.13:8777 check inter 2000 rise 2 fall 5
server controller3 10.0.0.14:8777 check inter 2000 rise 2 fall 5
listen nova_vncproxy_cluster
bind <Virtual IP>:6080
balance source
option tcpka
option tcplog
server controller1 10.0.0.12:6080 check inter 2000 rise 2 fall 5
server controller2 10.0.0.13:6080 check inter 2000 rise 2 fall 5
server controller3 10.0.0.14:6080 check inter 2000 rise 2 fall 5
listen neutron_api_cluster
bind <Virtual IP>:9696
balance source
option tcpka
option httpchk
option tcplog
server controller1 10.0.0.12:9696 check inter 2000 rise 2 fall 5
server controller2 10.0.0.13:9696 check inter 2000 rise 2 fall 5
server controller3 10.0.0.14:9696 check inter 2000 rise 2 fall 5
listen swift_proxy_cluster
bind <Virtual IP>:8080
balance source
option tcplog
option tcpka
server controller1 10.0.0.12:8080 check inter 2000 rise 2 fall 5
server controller2 10.0.0.13:8080 check inter 2000 rise 2 fall 5
server controller3 10.0.0.14:8080 check inter 2000 rise 2 fall 5
.. note::
The Galera cluster configuration directive ``backup`` indicates
that two of the three controllers are standby nodes.
This ensures that only one node services write requests
because OpenStack support for multi-node writes is not yet production-ready.
.. note::
The Telemetry API service configuration does not have the ``option httpchk``
directive as it cannot process this check properly.
.. TODO: explain why the Telemetry API is so special
#. Configure the kernel parameter to allow non-local IP binding. This allows
running HAProxy instances to bind to a VIP for failover. Add following line
to ``/etc/sysctl.conf``:
.. code-block:: none
net.ipv4.ip_nonlocal_bind = 1
#. Restart the host or, to make changes work immediately, invoke:
.. code-block:: console
$ sysctl -p
#. Add HAProxy to the cluster and ensure the VIPs can only run on machines
where HAProxy is active:
``pcs``
.. code-block:: console
$ pcs resource create lb-haproxy systemd:haproxy --clone
$ pcs constraint order start vip then lb-haproxy-clone kind=Optional
$ pcs constraint colocation add lb-haproxy-clone with vip
``crmsh``
.. code-block:: console
$ crm cib new conf-haproxy
$ crm configure primitive haproxy lsb:haproxy op monitor interval="1s"
$ crm configure clone haproxy-clone haproxy
$ crm configure colocation vip-with-haproxy inf: vip haproxy-clone
$ crm configure order haproxy-after-vip mandatory: vip haproxy-clone

167
doc/ha-guide/source/controller-ha-identity.rst
View File

@ -1,167 +0,0 @@
=============================
Highly available Identity API
=============================
Making the OpenStack Identity service highly available
in active and passive mode involves:
- :ref:`identity-pacemaker`
- :ref:`identity-config-identity`
- :ref:`identity-services-config`
.. _identity-pacemaker:
Prerequisites
~~~~~~~~~~~~~
Before beginning, ensure you have read the
`OpenStack Identity service getting started documentation
<https://docs.openstack.org/admin-guide/common/get-started-identity.html>`_.
Add OpenStack Identity resource to Pacemaker
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The following section(s) detail how to add the OpenStack Identity
resource to Pacemaker on SUSE and Red Hat.
SUSE
-----
SUSE Enterprise Linux and SUSE-based distributions, such as openSUSE,
use a set of OCF agents for controlling OpenStack services.
#. Run the following commands to download the OpenStack Identity resource
to Pacemaker:
.. code-block:: console
# cd /usr/lib/ocf/resource.d
# mkdir openstack
# cd openstack
# wget https://git.openstack.org/cgit/openstack/openstack-resource-agents/plain/ocf/keystone
# chmod a+rx *
#. Add the Pacemaker configuration for the OpenStack Identity resource
by running the following command to connect to the Pacemaker cluster:
.. code-block:: console
# crm configure
#. Add the following cluster resources:
.. code-block:: console
clone p_keystone ocf:openstack:keystone \
params config="/etc/keystone/keystone.conf" os_password="secretsecret" os_username="admin" os_tenant_name="admin" os_auth_url="http://10.0.0.11:5000/v2.0/" \
op monitor interval="30s" timeout="30s"
.. note::
This configuration creates ``p_keystone``,
a resource for managing the OpenStack Identity service.
#. Commit your configuration changes from the :command:`crm configure` menu
with the following command:
.. code-block:: console
# commit
The :command:`crm configure` supports batch input. You may have to copy and
paste the above lines into your live Pacemaker configuration, and then make
changes as required.
For example, you may enter ``edit p_ip_keystone`` from the
:command:`crm configure` menu and edit the resource to match your preferred
virtual IP address.
Pacemaker now starts the OpenStack Identity service and its dependent
resources on all of your nodes.
Red Hat
--------
For Red Hat Enterprise Linux and Red Hat-based Linux distributions,
the following process uses Systemd unit files.
.. code-block:: console
# pcs resource create openstack-keystone systemd:openstack-keystone --clone interleave=true
.. _identity-config-identity:
Configure OpenStack Identity service
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#. Edit the :file:`keystone.conf` file
to change the values of the :manpage:`bind(2)` parameters:
.. code-block:: ini
bind_host = 10.0.0.12
public_bind_host = 10.0.0.12
admin_bind_host = 10.0.0.12
The ``admin_bind_host`` parameter
lets you use a private network for admin access.
#. To be sure that all data is highly available,
ensure that everything is stored in the MySQL database
(which is also highly available):
.. code-block:: ini
[catalog]
driver = keystone.catalog.backends.sql.Catalog
# ...
[identity]
driver = keystone.identity.backends.sql.Identity
# ...
#. If the Identity service will be sending ceilometer notifications
and your message bus is configured for high availability, you will
need to ensure that the Identity service is correctly configured to
use it. For details on how to configure the Identity service for
this kind of deployment, see :doc:`shared-messaging`.
.. _identity-services-config:
Configure OpenStack services to use the highly available OpenStack Identity
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Your OpenStack services now point their OpenStack Identity configuration
to the highly available virtual cluster IP address.
#. For OpenStack Compute, (if your OpenStack Identity service IP address
is 10.0.0.11) use the following configuration in the :file:`api-paste.ini`
file:
.. code-block:: ini
auth_host = 10.0.0.11
#. Create the OpenStack Identity Endpoint with this IP address.
.. note::
If you are using both private and public IP addresses,
create two virtual IP addresses and define the endpoint. For
example:
.. code-block:: console
$ openstack endpoint create --region $KEYSTONE_REGION \
$service-type public http://PUBLIC_VIP:5000/v2.0
$ openstack endpoint create --region $KEYSTONE_REGION \
$service-type admin http://10.0.0.11:35357/v2.0
$ openstack endpoint create --region $KEYSTONE_REGION \
$service-type internal http://10.0.0.11:5000/v2.0
#. If you are using the horizon Dashboard, edit the :file:`local_settings.py`
file to include the following:
.. code-block:: ini
OPENSTACK_HOST = 10.0.0.11

21
doc/ha-guide/source/controller-ha-memcached.rst
View File

@ -1,21 +0,0 @@
=========
Memcached
=========
Memcached is a general-purpose distributed memory caching system. It
is used to speed up dynamic database-driven websites by caching data
and objects in RAM to reduce the number of times an external data
source must be read.
Memcached is a memory cache demon that can be used by most OpenStack
services to store ephemeral data, such as tokens.
Access to Memcached is not handled by HAProxy because replicated
access is currently in an experimental state. Instead, OpenStack
services must be supplied with the full list of hosts running
Memcached.
The Memcached client implements hashing to balance objects among the
instances. Failure of an instance impacts only a percentage of the
objects and the client automatically removes it from the list of
instances. The SLA is several minutes.

631
doc/ha-guide/source/controller-ha-pacemaker.rst
View File

@ -1,631 +0,0 @@
=======================
Pacemaker cluster stack
=======================
`Pacemaker <http://clusterlabs.org/>`_ cluster stack is a state-of-the-art
high availability and load balancing stack for the Linux platform.
Pacemaker is used to make OpenStack infrastructure highly available.
.. note::
It is storage and application-agnostic, and in no way specific to OpenStack.
Pacemaker relies on the
`Corosync <https://corosync.github.io/corosync/>`_ messaging layer
for reliable cluster communications. Corosync implements the Totem single-ring
ordering and membership protocol. It also provides UDP and InfiniBand based
messaging, quorum, and cluster membership to Pacemaker.
Pacemaker does not inherently understand the applications it manages.
Instead, it relies on resource agents (RAs) that are scripts that encapsulate
the knowledge of how to start, stop, and check the health of each application
managed by the cluster.
These agents must conform to one of the `OCF <https://github.com/ClusterLabs/
OCF-spec/blob/master/ra/resource-agent-api.md>`_,
`SysV Init <http://refspecs.linux-foundation.org/LSB_3.0.0/LSB-Core-generic/
LSB-Core-generic/iniscrptact.html>`_, Upstart, or Systemd standards.
Pacemaker ships with a large set of OCF agents (such as those managing
MySQL databases, virtual IP addresses, and RabbitMQ), but can also use
any agents already installed on your system and can be extended with
your own (see the
`developer guide <http://www.linux-ha.org/doc/dev-guides/ra-dev-guide.html>`_).
The steps to implement the Pacemaker cluster stack are:
- :ref:`pacemaker-install`
- :ref:`pacemaker-corosync-setup`
- :ref:`pacemaker-corosync-start`
- :ref:`pacemaker-start`
- :ref:`pacemaker-cluster-properties`
.. _pacemaker-install:
Install packages
~~~~~~~~~~~~~~~~
On any host that is meant to be part of a Pacemaker cluster, establish cluster
communications through the Corosync messaging layer.
This involves installing the following packages (and their dependencies, which
your package manager usually installs automatically):
- `pacemaker`
- `pcs` (CentOS or RHEL) or crmsh
- `corosync`
- `fence-agents` (CentOS or RHEL) or cluster-glue
- `resource-agents`
- `libqb0`
.. _pacemaker-corosync-setup:
Set up the cluster with pcs
~~~~~~~~~~~~~~~~~~~~~~~~~~~
#. Make sure `pcs` is running and configured to start at boot time:
.. code-block:: console
$ systemctl enable pcsd
$ systemctl start pcsd
#. Set a password for hacluster user on each host:
.. code-block:: console
$ echo my-secret-password-no-dont-use-this-one \
| passwd --stdin hacluster
.. note::
Since the cluster is a single administrative domain, it is
acceptable to use the same password on all nodes.
#. Use that password to authenticate to the nodes that will
make up the cluster:
.. code-block:: console
$ pcs cluster auth controller1 controller2 controller3 \
-u hacluster -p my-secret-password-no-dont-use-this-one --force
.. note::
The ``-p`` option is used to give the password on command
line and makes it easier to script.
#. Create and name the cluster. Then, start it and enable all components to
auto-start at boot time:
.. code-block:: console
$ pcs cluster setup --force --name my-first-openstack-cluster \
controller1 controller2 controller3
$ pcs cluster start --all
$ pcs cluster enable --all
.. note ::
In Red Hat Enterprise Linux or CentOS environments, this is a recommended
path to perform configuration. For more information, see the `RHEL docs
<https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/High_Availability_Add-On_Reference/ch-clusteradmin-HAAR.html#s1-clustercreate-HAAR>`_.
Set up the cluster with `crmsh`
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
After installing the Corosync package, you must create
the :file:`/etc/corosync/corosync.conf` configuration file.
.. note::
For Ubuntu, you should also enable the Corosync service in the
``/etc/default/corosync`` configuration file.
Corosync can be configured to work with either multicast or unicast IP
addresses or to use the votequorum library.
- :ref:`corosync-multicast`
- :ref:`corosync-unicast`
- :ref:`corosync-votequorum`
.. _corosync-multicast:
Set up Corosync with multicast
------------------------------
Most distributions ship an example configuration file
(:file:`corosync.conf.example`) as part of the documentation bundled with
the Corosync package. An example Corosync configuration file is shown below:
**Example Corosync configuration file for multicast (``corosync.conf``)**
.. code-block:: none
totem {
version: 2
# Time (in ms) to wait for a token (1)
token: 10000
# How many token retransmits before forming a new
# configuration
token_retransmits_before_loss_const: 10
# Turn off the virtual synchrony filter
vsftype: none
# Enable encryption (2)
secauth: on
# How many threads to use for encryption/decryption
threads: 0
# This specifies the redundant ring protocol, which may be
# none, active, or passive. (3)
rrp_mode: active
# The following is a two-ring multicast configuration. (4)
interface {
ringnumber: 0
bindnetaddr: 10.0.0.0
mcastaddr: 239.255.42.1
mcastport: 5405
}
interface {
ringnumber: 1
bindnetaddr: 10.0.42.0
mcastaddr: 239.255.42.2
mcastport: 5405
}
}
amf {
mode: disabled
}
service {
# Load the Pacemaker Cluster Resource Manager (5)
ver: 1
name: pacemaker
}
aisexec {
user: root
group: root
}
logging {
fileline: off
to_stderr: yes
to_logfile: no
to_syslog: yes
syslog_facility: daemon
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
tags: enter|leave|trace1|trace2|trace3|trace4|trace6
}}
Note the following:
- The ``token`` value specifies the time, in milliseconds,
during which the Corosync token is expected
to be transmitted around the ring.
When this timeout expires, the token is declared lost,
and after ``token_retransmits_before_loss_const lost`` tokens,
the non-responding processor (cluster node) is declared dead.
``token × token_retransmits_before_loss_const``
is the maximum time a node is allowed to not respond to cluster messages
before being considered dead.
The default for token is 1000 milliseconds (1 second),
with 4 allowed retransmits.
These defaults are intended to minimize failover times,
but can cause frequent false alarms and unintended failovers
in case of short network interruptions. The values used here are safer,
albeit with slightly extended failover times.
- With ``secauth`` enabled,
Corosync nodes mutually authenticates using a 128-byte shared secret
stored in the :file:`/etc/corosync/authkey` file.
This can be generated with the :command:`corosync-keygen` utility.
Cluster communications are encrypted when using ``secauth``.
- In Corosync, configurations use redundant networking
(with more than one interface). This means you must select a Redundant
Ring Protocol (RRP) mode other than none. We recommend ``active`` as
the RRP mode.
Note the following about the recommended interface configuration:
- Each configured interface must have a unique ``ringnumber``,
starting with 0.
- The ``bindnetaddr`` is the network address of the interfaces to bind to.
The example uses two network addresses of /24 IPv4 subnets.
- Multicast groups (``mcastaddr``) must not be reused
across cluster boundaries. No two distinct clusters
should ever use the same multicast group.
Be sure to select multicast addresses compliant with
`RFC 2365, "Administratively Scoped IP Multicast"
<http://www.ietf.org/rfc/rfc2365.txt>`_.
- For firewall configurations, Corosync communicates over UDP only,
and uses ``mcastport`` (for receives) and ``mcastport - 1`` (for sends).
- The service declaration for the Pacemaker service
may be placed in the :file:`corosync.conf` file directly
or in its own separate file, :file:`/etc/corosync/service.d/pacemaker`.
.. note::
If you are using Corosync version 2 on Ubuntu 14.04,
remove or comment out lines under the service stanza.
These stanzas enable Pacemaker to start up. Another potential
problem is the boot and shutdown order of Corosync and
Pacemaker. To force Pacemaker to start after Corosync and
stop before Corosync, fix the start and kill symlinks manually:
.. code-block:: console
# update-rc.d pacemaker start 20 2 3 4 5 . stop 00 0 1 6 .
The Pacemaker service also requires an additional
configuration file ``/etc/corosync/uidgid.d/pacemaker``
to be created with the following content:
.. code-block:: none
uidgid {
uid: hacluster
gid: haclient
}
- Once created, synchronize the :file:`corosync.conf` file
(and the :file:`authkey` file if the secauth option is enabled)
across all cluster nodes.
.. _corosync-unicast:
Set up Corosync with unicast
----------------------------
For environments that do not support multicast, Corosync should be configured
for unicast. An example fragment of the :file:`corosync.conf` file
for unicastis is shown below:
**Corosync configuration file fragment for unicast (``corosync.conf``)**
.. code-block:: none
totem {
#...
interface {
ringnumber: 0
bindnetaddr: 10.0.0.0
broadcast: yes (1)
mcastport: 5405
}
interface {
ringnumber: 1
bindnetaddr: 10.0.42.0
broadcast: yes
mcastport: 5405
}
transport: udpu (2)
}
nodelist { (3)
node {
ring0_addr: 10.0.0.12
ring1_addr: 10.0.42.12
nodeid: 1
}
node {
ring0_addr: 10.0.0.13
ring1_addr: 10.0.42.13
nodeid: 2
}
node {
ring0_addr: 10.0.0.14
ring1_addr: 10.0.42.14
nodeid: 3
}
}
#...
Note the following:
- If the ``broadcast`` parameter is set to ``yes``, the broadcast address is
used for communication. If this option is set, the ``mcastaddr`` parameter
should not be set.
- The ``transport`` directive controls the transport mechanism.
To avoid the use of multicast entirely, specify the ``udpu`` unicast
transport parameter. This requires specifying the list of members in the
``nodelist`` directive. This potentially makes up the membership before
deployment. The default is ``udp``. The transport type can also be set to
``udpu`` or ``iba``.
- Within the ``nodelist`` directive, it is possible to specify specific
information about the nodes in the cluster. The directive can contain only
the node sub-directive, which specifies every node that should be a member
of the membership, and where non-default options are needed. Every node must
have at least the ``ring0_addr`` field filled.
.. note::
For UDPU, every node that should be a member of the membership must be specified.
Possible options are:
- ``ring{X}_addr`` specifies the IP address of one of the nodes.
``{X}`` is the ring number.
- ``nodeid`` is optional when using IPv4 and required when using IPv6.
This is a 32-bit value specifying the node identifier delivered to the
cluster membership service. If this is not specified with IPv4,
the node ID is determined from the 32-bit IP address of the system to which
the system is bound with ring identifier of 0. The node identifier value of
zero is reserved and should not be used.
.. _corosync-votequorum:
Set up Corosync with votequorum library
---------------------------------------
The votequorum library is part of the Corosync project. It provides an
interface to the vote-based quorum service and it must be explicitly enabled
in the Corosync configuration file. The main role of votequorum library is to
avoid split-brain situations, but it also provides a mechanism to:
- Query the quorum status
- List the nodes known to the quorum service
- Receive notifications of quorum state changes
- Change the number of votes assigned to a node
- Change the number of expected votes for a cluster to be quorate
- Connect an additional quorum device to allow small clusters remain quorate
during node outages
The votequorum library has been created to replace and eliminate ``qdisk``, the
disk-based quorum daemon for CMAN, from advanced cluster configurations.
A sample votequorum service configuration in the :file:`corosync.conf` file is:
.. code-block:: none
quorum {
provider: corosync_votequorum (1)
expected_votes: 7 (2)
wait_for_all: 1 (3)
last_man_standing: 1 (4)
last_man_standing_window: 10000 (5)
}
Note the following:
- Specifying ``corosync_votequorum`` enables the votequorum library.
This is the only required option.
- The cluster is fully operational with ``expected_votes`` set to 7 nodes
(each node has 1 vote), quorum: 4. If a list of nodes is specified as
``nodelist``, the ``expected_votes`` value is ignored.
- When you start up a cluster (all nodes down) and set ``wait_for_all`` to 1,
the cluster quorum is held until all nodes are online and have joined the
cluster for the first time. This parameter is new in Corosync 2.0.
- Setting ``last_man_standing`` to 1 enables the Last Man Standing (LMS)
feature. By default, it is disabled (set to 0).
If a cluster is on the quorum edge (``expected_votes:`` set to 7;
``online nodes:`` set to 4) for longer than the time specified
for the ``last_man_standing_window`` parameter, the cluster can recalculate
quorum and continue operating even if the next node will be lost.
This logic is repeated until the number of online nodes in the cluster
reaches 2. In order to allow the cluster to step down from 2 members to only
1, the ``auto_tie_breaker`` parameter needs to be set.
We do not recommended this for production environments.
- ``last_man_standing_window`` specifies the time, in milliseconds,
required to recalculate quorum after one or more hosts
have been lost from the cluster. To perform a new quorum recalculation,
the cluster must have quorum for at least the interval
specified for ``last_man_standing_window``. The default is 10000ms.
.. _pacemaker-corosync-start:
Start Corosync
--------------
Corosync is started as a regular system service. Depending on your
distribution, it may ship with an LSB init script, an upstart job, or
a Systemd unit file.
- Start ``corosync`` with the LSB init script:
.. code-block:: console
# /etc/init.d/corosync start
Alternatively:
.. code-block:: console
# service corosync start
- Start ``corosync`` with upstart:
.. code-block:: console
# start corosync
- Start ``corosync`` with systemd unit file:
.. code-block:: console
# systemctl start corosync
You can now check the ``corosync`` connectivity with one of these tools.
Use the :command:`corosync-cfgtool` utility with the ``-s`` option
to get a summary of the health of the communication rings:
.. code-block:: console
# corosync-cfgtool -s
Printing ring status.
Local node ID 435324542
RING ID 0
id = 10.0.0.82
status = ring 0 active with no faults
RING ID 1
id = 10.0.42.100
status = ring 1 active with no faults
Use the :command:`corosync-objctl` utility to dump the Corosync cluster
member list:
.. note::
If you are using Corosync version 2, use the :command:`corosync-cmapctl`
utility instead of :command:`corosync-objctl`; it is a direct replacement.
.. code-block:: console
# corosync-objctl runtime.totem.pg.mrp.srp.members
runtime.totem.pg.mrp.srp.435324542.ip=r(0) ip(10.0.0.82) r(1) ip(10.0.42.100)
runtime.totem.pg.mrp.srp.435324542.join_count=1
runtime.totem.pg.mrp.srp.435324542.status=joined
runtime.totem.pg.mrp.srp.983895584.ip=r(0) ip(10.0.0.87) r(1) ip(10.0.42.254)
runtime.totem.pg.mrp.srp.983895584.join_count=1
runtime.totem.pg.mrp.srp.983895584.status=joined
You should see a ``status=joined`` entry for each of your constituent
cluster nodes.
.. _pacemaker-start:
Start Pacemaker
---------------
After the ``corosync`` service have been started and you have verified that the
cluster is communicating properly, you can start :command:`pacemakerd`, the
Pacemaker master control process. Choose one from the following four ways to
start it:
#. Start ``pacemaker`` with the LSB init script:
.. code-block:: console
# /etc/init.d/pacemaker start
Alternatively:
.. code-block:: console
# service pacemaker start
#. Start ``pacemaker`` with upstart:
.. code-block:: console
# start pacemaker
#. Start ``pacemaker`` with the systemd unit file:
.. code-block:: console
# systemctl start pacemaker
After the ``pacemaker`` service has started, Pacemaker creates a default empty
cluster configuration with no resources. Use the :command:`crm_mon` utility to
observe the status of ``pacemaker``:
.. code-block:: console
# crm_mon -1
Last updated: Sun Oct 7 21:07:52 2012
Last change: Sun Oct 7 20:46:00 2012 via cibadmin on controller2
Stack: openais
Current DC: controller2 - partition with quorum
Version: 1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c
3 Nodes configured, 3 expected votes
0 Resources configured.
Online: [ controller3 controller2 controller1 ]
...
.. _pacemaker-cluster-properties:
Set basic cluster properties
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
After you set up your Pacemaker cluster, set a few basic cluster properties:
- ``crmsh``
.. code-block:: console
$ crm configure property pe-warn-series-max="1000" \
pe-input-series-max="1000" \
pe-error-series-max="1000" \
cluster-recheck-interval="5min"
- ``pcs``
.. code-block:: console
$ pcs property set pe-warn-series-max=1000 \
pe-input-series-max=1000 \
pe-error-series-max=1000 \
cluster-recheck-interval=5min
Note the following:
- Setting the ``pe-warn-series-max``, ``pe-input-series-max``,
and ``pe-error-series-max`` parameters to 1000
instructs Pacemaker to keep a longer history of the inputs processed
and errors and warnings generated by its Policy Engine.
This history is useful if you need to troubleshoot the cluster.
- Pacemaker uses an event-driven approach to cluster state processing.
The ``cluster-recheck-interval`` parameter (which defaults to 15 minutes)
defines the interval at which certain Pacemaker actions occur.
It is usually prudent to reduce this to a shorter interval,
such as 5 or 3 minutes.
By default, STONITH is enabled in Pacemaker, but STONITH mechanisms (to
shutdown a node via IPMI or ssh) are not configured. In this case Pacemaker
will refuse to start any resources.
For production cluster it is recommended to configure appropriate STONITH
mechanisms. But for demo or testing purposes STONITH can be disabled completely
as follows:
- ``crmsh``
.. code-block:: console
$ crm configure property stonith-enabled=false
- ``pcs``
.. code-block:: console
$ pcs property set stonith-enabled=false
After you make these changes, commit the updated configuration.

80
doc/ha-guide/source/controller-ha-telemetry.rst
View File

@ -1,80 +0,0 @@
==========================
Highly available Telemetry
==========================
The `Telemetry service
<https://docs.openstack.org/admin-guide/common/get-started-telemetry.html>`_
provides a data collection service and an alarming service.
Telemetry polling agent
~~~~~~~~~~~~~~~~~~~~~~~
The Telemetry polling agent can be configured to partition its polling
workload between multiple agents. This enables high availability (HA).
Both the central and the compute agent can run in an HA deployment.
This means that multiple instances of these services can run in
parallel with workload partitioning among these running instances.
The `Tooz <https://pypi.org/project/tooz>`_ library provides
the coordination within the groups of service instances.
It provides an API above several back ends that can be used for building
distributed applications.
Tooz supports
`various drivers <https://docs.openstack.org/tooz/latest/user/drivers.html>`_
including the following back end solutions:
* `Zookeeper <https://zookeeper.apache.org/>`_:
Recommended solution by the Tooz project.
* `Redis <https://redis.io/>`_:
Recommended solution by the Tooz project.
* `Memcached <https://memcached.org/>`_:
Recommended for testing.
You must configure a supported Tooz driver for the HA deployment of
the Telemetry services.
For information about the required configuration options
to set in the :file:`ceilometer.conf`, see the `coordination section
<https://docs.openstack.org/ocata/config-reference/telemetry.html>`_
in the OpenStack Configuration Reference.
.. note::
Only one instance for the central and compute agent service(s) is able
to run and function correctly if the ``backend_url`` option is not set.
The availability check of the instances is provided by heartbeat messages.
When the connection with an instance is lost, the workload will be
reassigned within the remaining instances in the next polling cycle.
.. note::
Memcached uses a timeout value, which should always be set to
a value that is higher than the heartbeat value set for Telemetry.
For backward compatibility and supporting existing deployments, the central
agent configuration supports using different configuration files. This is for
groups of service instances that are running in parallel.
For enabling this configuration, set a value for the
``partitioning_group_prefix`` option in the
`polling section <https://docs.openstack.org/ocata/config-reference/telemetry/telemetry-config-options.html>`_
in the OpenStack Configuration Reference.
.. warning::
For each sub-group of the central agent pool with the same
``partitioning_group_prefix``, a disjoint subset of meters must be polled
to avoid samples being missing or duplicated. The list of meters to poll
can be set in the :file:`/etc/ceilometer/pipeline.yaml` configuration file.
For more information about pipelines see the `Data processing and pipelines
<https://docs.openstack.org/admin-guide/telemetry-data-pipelines.html>`_
section.
To enable the compute agent to run multiple instances simultaneously with
workload partitioning, the ``workload_partitioning`` option must be set to
``True`` under the `compute section <https://docs.openstack.org/ocata/config-reference/telemetry.html>`_
in the :file:`ceilometer.conf` configuration file.

23
doc/ha-guide/source/controller-ha-vip.rst
View File

@ -1,23 +0,0 @@
=================
Configure the VIP
=================
You must select and assign a virtual IP address (VIP) that can freely float
between cluster nodes.
This configuration creates ``vip``, a virtual IP address for use by the
API node (``10.0.0.11``).
For ``crmsh``:
.. code-block:: console
# crm configure primitive vip ocf:heartbeat:IPaddr2 \
params ip="10.0.0.11" cidr_netmask="24" op monitor interval="30s"
For ``pcs``:
.. code-block:: console
# pcs resource create vip ocf:heartbeat:IPaddr2 \
ip="10.0.0.11" cidr_netmask="24" op monitor interval="30s"

73
doc/ha-guide/source/controller-ha.rst
View File

@ -1,73 +0,0 @@
==========================
Configuring the controller
==========================
The cloud controller runs on the management network and must talk to
all other services.
.. toctree::
:maxdepth: 2
intro-ha-arch-pacemaker.rst
controller-ha-pacemaker.rst
controller-ha-vip.rst
controller-ha-haproxy.rst
controller-ha-memcached.rst
controller-ha-identity.rst
controller-ha-telemetry.rst
Overview of highly available controllers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
OpenStack is a set of services exposed to the end users
as HTTP(s) APIs. Additionally, for your own internal usage, OpenStack
requires an SQL database server and AMQP broker. The physical servers,
where all the components are running, are called controllers.
This modular OpenStack architecture allows you to duplicate all the
components and run them on different controllers.
By making all the components redundant, it is possible to make
OpenStack highly available.
In general, we can divide all the OpenStack components into three categories:
- OpenStack APIs: APIs that are HTTP(s) stateless services written in python,
easy to duplicate and mostly easy to load balance.
- The SQL relational database server provides stateful type consumed by other
components. Supported databases are MySQL, MariaDB, and PostgreSQL.
Making the SQL database redundant is complex.
- :term:`Advanced Message Queuing Protocol (AMQP)` provides OpenStack
internal stateful communication service.
Common deployment architectures
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
We recommend two primary architectures for making OpenStack highly available.
The architectures differ in the sets of services managed by the
cluster.
Both use a cluster manager, such as Pacemaker or Veritas, to
orchestrate the actions of the various services across a set of
machines. Because we are focused on FOSS, we refer to these as
Pacemaker architectures.
Traditionally, Pacemaker has been positioned as an all-encompassing
solution. However, as OpenStack services have matured, they are
increasingly able to run in an active/active configuration and
gracefully tolerate the disappearance of the APIs on which they
depend.
With this in mind, some vendors are restricting Pacemaker's use to
services that must operate in an active/passive mode (such as
``cinder-volume``), those with multiple states (for example, Galera), and
those with complex bootstrapping procedures (such as RabbitMQ).
The majority of services, needing no real orchestration, are handled
by systemd on each node. This approach avoids the need to coordinate
service upgrades or location changes with the cluster and has the
added advantage of more easily scaling beyond Corosync's 16 node
limit. However, it will generally require the addition of an
enterprise monitoring solution such as Nagios or Sensu for those
wanting centralized failure reporting.

61
doc/ha-guide/source/environment-hardware.rst
View File

@ -1,61 +0,0 @@
=============================================
Hardware considerations for high availability
=============================================
When you use high availability, consider the hardware requirements needed
for your application.
Hardware setup
~~~~~~~~~~~~~~
The following are the standard hardware requirements:
- Provider networks: See the *Overview -> Networking Option 1: Provider
networks* section of the
`Install Guides <https://docs.openstack.org/ocata/install>`_
depending on your distribution.
- Self-service networks: See the *Overview -> Networking Option 2:
Self-service networks* section of the
`Install Guides <https://docs.openstack.org/ocata/install>`_
depending on your distribution.
OpenStack does not require a significant amount of resources and the following
minimum requirements should support a proof-of-concept high availability
environment with core services and several instances:
+-------------------+------------------+----------+-----------+------+
| Node type | Processor Cores | Memory | Storage | NIC |
+===================+==================+==========+===========+======+
| controller node | 4 | 12 GB | 120 GB | 2 |
+-------------------+------------------+----------+-----------+------+
| compute node | 8+ | 12+ GB | 120+ GB | 2 |
+-------------------+------------------+----------+-----------+------+
We recommended that the maximum latency between any two controller
nodes is 2 milliseconds. Although the cluster software can be tuned to
operate at higher latencies, some vendors insist on this value before
agreeing to support the installation.
You can use the `ping` command to find the latency between two servers.
Virtualized hardware
~~~~~~~~~~~~~~~~~~~~
For demonstrations and studying, you can set up a test environment on virtual
machines (VMs). This has the following benefits:
- One physical server can support multiple nodes,
each of which supports almost any number of network interfaces.
- You can take periodic snap shots throughout the installation process
and roll back to a working configuration in the event of a problem.
However, running an OpenStack environment on VMs degrades the performance of
your instances, particularly if your hypervisor or processor lacks
support for hardware acceleration of nested VMs.
.. note::
When installing highly available OpenStack on VMs,
be sure that your hypervisor permits promiscuous mode
and disables MAC address filtering on the external network.

32
doc/ha-guide/source/environment-memcached.rst
View File

@ -1,32 +0,0 @@
====================
Installing Memcached
====================
Most OpenStack services can use Memcached to store ephemeral data such as
tokens. Although Memcached does not support typical forms of redundancy such
as clustering, OpenStack services can use almost any number of instances
by configuring multiple hostnames or IP addresses.
The Memcached client implements hashing to balance objects among the instances.
Failure of an instance only impacts a percentage of the objects,
and the client automatically removes it from the list of instances.
To install and configure Memcached, read the
`official documentation <https://github.com/Memcached/Memcached/wiki#getting-started>`_.
Memory caching is managed by `oslo.cache
<http://specs.openstack.org/openstack/oslo-specs/specs/kilo/oslo-cache-using-dogpile.html>`_.
This ensures consistency across all projects when using multiple Memcached
servers. The following is an example configuration with three hosts:
.. code-block:: ini
Memcached_servers = controller1:11211,controller2:11211,controller3:11211
By default, ``controller1`` handles the caching service. If the host goes down,
``controller2`` or ``controller3`` will complete the service.
For more information about Memcached installation, see the
*Environment -> Memcached* section in the
`Installation Guides <https://docs.openstack.org/ocata/install/>`_
depending on your distribution.

10
doc/ha-guide/source/environment-ntp.rst
View File

@ -1,10 +0,0 @@
=============
Configure NTP
=============
You must configure NTP to properly synchronize services among nodes.
We recommend that you configure the controller node to reference
more accurate (lower stratum) servers and other nodes to reference
the controller node. For more information, see the
`Installation Guides
<https://docs.openstack.org/ocata/install/>`_.

29
doc/ha-guide/source/environment-operatingsystem.rst
View File

@ -1,29 +0,0 @@
===============================
Installing the operating system
===============================
The first step in setting up your highly available OpenStack cluster
is to install the operating system on each node.
Follow the instructions in the *Environment* section of the
`Installation Guides <https://docs.openstack.org/ocata/install>`_
depending on your distribution.
The OpenStack Installation Guides also include a list of
the services that use passwords with important notes about using
them.
.. note::
Before following this guide to configure the highly available
OpenStack cluster, ensure the IP ``10.0.0.11`` and hostname
``controller`` are not in use.
This guide uses the following example IP addresses:
.. code-block:: none
# controller
10.0.0.11 controller # virtual IP
10.0.0.12 controller1
10.0.0.13 controller2
10.0.0.14 controller3

14
doc/ha-guide/source/environment.rst
View File

@ -1,14 +0,0 @@
=================================
Configuring the basic environment
=================================
This chapter describes the basic environment for high availability,
such as hardware, operating system, common services.
.. toctree::
:maxdepth: 2
environment-hardware.rst
environment-operatingsystem.rst
environment-ntp.rst
environment-memcached.rst

BIN
doc/ha-guide/source/figures/Cluster-deployment-collapsed.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 223 KiB

BIN
doc/ha-guide/source/figures/Cluster-deployment-segregated.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 215 KiB

50
doc/ha-guide/source/index.rst
View File

@ -1,50 +0,0 @@
=================================
OpenStack High Availability Guide
=================================
Abstract
~~~~~~~~
This guide describes how to install and configure OpenStack for high
availability. It supplements the Installation Guides
and assumes that you are familiar with the material in those guides.
.. important::
This guide was last updated as of the Ocata release, documenting
the OpenStack Ocata, Newton, and Mitaka releases. It may
not apply to EOL releases Kilo and Liberty.
We advise that you read this at your own discretion when planning
on your OpenStack cloud.
This guide is intended as advice only.
The OpenStack HA team is based on voluntary contributions from
the OpenStack community. You can contact the HA community
directly in the #openstack-ha channel on Freenode IRC, or by
sending mail to the openstack-dev mailing list with the [HA] prefix in
the subject header.
The OpenStack HA community used to hold `weekly IRC meetings
<https://wiki.openstack.org/wiki/Meetings/HATeamMeeting>`_ to discuss
a range of topics relating to HA in OpenStack. The
`logs of all past meetings
<http://eavesdrop.openstack.org/meetings/ha/>`_ are still available to
read.
Contents
~~~~~~~~
.. toctree::
:maxdepth: 2
common/conventions.rst
intro-ha.rst
environment.rst
shared-services.rst
controller-ha.rst
networking-ha.rst
storage-ha.rst
compute-node-ha.rst
appendix.rst

190
doc/ha-guide/source/intro-ha-arch-pacemaker.rst
View File

@ -1,190 +0,0 @@
==========================
The Pacemaker architecture
==========================
What is a cluster manager?
~~~~~~~~~~~~~~~~~~~~~~~~~~
At its core, a cluster is a distributed finite state machine capable
of co-ordinating the startup and recovery of inter-related services
across a set of machines.
Even a distributed or replicated application that is able to survive failures
on one or more machines can benefit from a cluster manager because a cluster
manager has the following capabilities:
#. Awareness of other applications in the stack
While SYS-V init replacements like systemd can provide
deterministic recovery of a complex stack of services, the
recovery is limited to one machine and lacks the context of what
is happening on other machines. This context is crucial to
determine the difference between a local failure, and clean startup
and recovery after a total site failure.
#. Awareness of instances on other machines
Services like RabbitMQ and Galera have complicated boot-up
sequences that require co-ordination, and often serialization, of
startup operations across all machines in the cluster. This is
especially true after a site-wide failure or shutdown where you must
first determine the last machine to be active.
#. A shared implementation and calculation of `quorum
<https://en.wikipedia.org/wiki/Quorum_(Distributed_Systems)>`_
It is very important that all members of the system share the same
view of who their peers are and whether or not they are in the
majority. Failure to do this leads very quickly to an internal
`split-brain <https://en.wikipedia.org/wiki/Split-brain_(computing)>`_
state. This is where different parts of the system are pulling in
different and incompatible directions.
#. Data integrity through fencing (a non-responsive process does not
imply it is not doing anything)
A single application does not have sufficient context to know the
difference between failure of a machine and failure of the
application on a machine. The usual practice is to assume the
machine is dead and continue working, however this is highly risky. A
rogue process or machine could still be responding to requests and
generally causing havoc. The safer approach is to make use of
remotely accessible power switches and/or network switches and SAN
controllers to fence (isolate) the machine before continuing.
#. Automated recovery of failed instances
While the application can still run after the failure of several
instances, it may not have sufficient capacity to serve the
required volume of requests. A cluster can automatically recover
failed instances to prevent additional load induced failures.
For these reasons, we highly recommend the use of a cluster manager like
`Pacemaker <http://clusterlabs.org>`_.
Deployment flavors
~~~~~~~~~~~~~~~~~~
It is possible to deploy three different flavors of the Pacemaker
architecture. The two extremes are ``Collapsed`` (where every
component runs on every node) and ``Segregated`` (where every
component runs in its own 3+ node cluster).
Regardless of which flavor you choose, we recommend that
clusters contain at least three nodes so that you can take advantage of
`quorum <quorum_>`_.
Quorum becomes important when a failure causes the cluster to split in
two or more partitions. In this situation, you want the majority members of
the system to ensure the minority are truly dead (through fencing) and continue
to host resources. For a two-node cluster, no side has the majority and
you can end up in a situation where both sides fence each other, or
both sides are running the same services. This can lead to data corruption.
Clusters with an even number of hosts suffer from similar issues. A
single network failure could easily cause a N:N split where neither
side retains a majority. For this reason, we recommend an odd number
of cluster members when scaling up.
You can have up to 16 cluster members (this is currently limited by
the ability of corosync to scale higher). In extreme cases, 32 and
even up to 64 nodes could be possible. However, this is not well tested.
Collapsed
---------
In a collapsed configuration, there is a single cluster of 3 or more
nodes on which every component is running.
This scenario has the advantage of requiring far fewer, if more
powerful, machines. Additionally, being part of a single cluster
allows you to accurately model the ordering dependencies between
components.
This scenario can be visualized as below.
.. image:: /figures/Cluster-deployment-collapsed.png
:width: 100%
You would choose this option if you prefer to have fewer but more
powerful boxes.
This is the most common option and the one we document here.
Segregated
----------
In this configuration, each service runs in a dedicated cluster of
3 or more nodes.
The benefits of this approach are the physical isolation between
components and the ability to add capacity to specific components.
You would choose this option if you prefer to have more but
less powerful boxes.
This scenario can be visualized as below, where each box below
represents a cluster of three or more guests.
.. image:: /figures/Cluster-deployment-segregated.png
:width: 100%
Mixed
-----
It is also possible to follow a segregated approach for one or more
components that are expected to be a bottleneck and use a collapsed
approach for the remainder.
Proxy server
~~~~~~~~~~~~
Almost all services in this stack benefit from being proxied.
Using a proxy server provides the following capabilities:
#. Load distribution
Many services can act in an active/active capacity, however, they
usually require an external mechanism for distributing requests to
one of the available instances. The proxy server can serve this
role.
#. API isolation
By sending all API access through the proxy, you can clearly
identify service interdependencies. You can also move them to
locations other than ``localhost`` to increase capacity if the
need arises.
#. Simplified process for adding/removing of nodes
Since all API access is directed to the proxy, adding or removing
nodes has no impact on the configuration of other services. This
can be very useful in upgrade scenarios where an entirely new set
of machines can be configured and tested in isolation before
telling the proxy to direct traffic there instead.
#. Enhanced failure detection
The proxy can be configured as a secondary mechanism for detecting
service failures. It can even be configured to look for nodes in
a degraded state (such as being too far behind in the
replication) and take them out of circulation.
The following components are currently unable to benefit from the use
of a proxy server:
* RabbitMQ
* Memcached
* MongoDB
We recommend HAProxy as the load balancer, however, there are many alternative
load balancing solutions in the marketplace.
Generally, we use round-robin to distribute load amongst instances of
active/active services. Alternatively, Galera uses ``stick-table`` options
to ensure that incoming connection to virtual IP (VIP) are directed to only one
of the available back ends. This helps avoid lock contention and prevent
deadlocks, although Galera can run active/active. Used in combination with
the ``httpchk`` option, this ensure only nodes that are in sync with their
peers are allowed to handle requests.

208
doc/ha-guide/source/intro-ha.rst
View File

@ -1,208 +0,0 @@
===========================================
Introduction to OpenStack high availability
===========================================
High availability systems seek to minimize the following issues:
#. System downtime: Occurs when a user-facing service is unavailable
beyond a specified maximum amount of time.
#. Data loss: Accidental deletion or destruction of data.
Most high availability systems guarantee protection against system downtime
and data loss only in the event of a single failure.
However, they are also expected to protect against cascading failures,
where a single failure deteriorates into a series of consequential failures.
Many service providers guarantee a :term:`Service Level Agreement (SLA)`
including uptime percentage of computing service, which is calculated based
on the available time and system downtime excluding planned outage time.
Redundancy and failover
~~~~~~~~~~~~~~~~~~~~~~~
High availability is implemented with redundant hardware
running redundant instances of each service.
If one piece of hardware running one instance of a service fails,
the system can then failover to use another instance of a service
that is running on hardware that did not fail.
A crucial aspect of high availability
is the elimination of single points of failure (SPOFs).
A SPOF is an individual piece of equipment or software
that causes system downtime or data loss if it fails.
In order to eliminate SPOFs, check that mechanisms exist for redundancy of:
- Network components, such as switches and routers
- Applications and automatic service migration
- Storage components
- Facility services such as power, air conditioning, and fire protection
In the event that a component fails and a back-up system must take on
its load, most high availability systems will replace the failed
component as quickly as possible to maintain necessary redundancy. This
way time spent in a degraded protection state is minimized.
Most high availability systems fail in the event of multiple
independent (non-consequential) failures. In this case, most
implementations favor protecting data over maintaining availability.
High availability systems typically achieve an uptime percentage of
99.99% or more, which roughly equates to less than an hour of
cumulative downtime per year. In order to achieve this, high
availability systems should keep recovery times after a failure to
about one to two minutes, sometimes significantly less.
OpenStack currently meets such availability requirements for its own
infrastructure services, meaning that an uptime of 99.99% is feasible
for the OpenStack infrastructure proper. However, OpenStack does not
guarantee 99.99% availability for individual guest instances.
This document discusses some common methods of implementing highly
available systems, with an emphasis on the core OpenStack services and
other open source services that are closely aligned with OpenStack.
You will need to address high availability concerns for any applications
software that you run on your OpenStack environment. The important thing is
to make sure that your services are redundant and available.
How you achieve that is up to you.
Stateless versus stateful services
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The following are the definitions of stateless and stateful services:
Stateless service
A service that provides a response after your request
and then requires no further attention.
To make a stateless service highly available,
you need to provide redundant instances and load balance them.
OpenStack services that are stateless include ``nova-api``,
``nova-conductor``, ``glance-api``, ``keystone-api``,
``neutron-api``, and ``nova-scheduler``.
Stateful service
A service where subsequent requests to the service
depend on the results of the first request.
Stateful services are more difficult to manage because a single
action typically involves more than one request. Providing
additional instances and load balancing does not solve the problem.
For example, if the horizon user interface reset itself every time
you went to a new page, it would not be very useful.
OpenStack services that are stateful include the OpenStack database
and message queue.
Making stateful services highly available can depend on whether you choose
an active/passive or active/active configuration.
Active/passive versus active/active
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Stateful services can be configured as active/passive or active/active,
which are defined as follows:
:term:`active/passive configuration`
Maintains a redundant instance
that can be brought online when the active service fails.
For example, OpenStack writes to the main database
while maintaining a disaster recovery database that can be brought online
if the main database fails.
A typical active/passive installation for a stateful service maintains
a replacement resource that can be brought online when required.
Requests are handled using a :term:`virtual IP address (VIP)` that
facilitates returning to service with minimal reconfiguration.
A separate application (such as Pacemaker or Corosync) monitors
these services, bringing the backup online as necessary.
:term:`active/active configuration`
Each service also has a backup but manages both the main and
redundant systems concurrently.
This way, if there is a failure, the user is unlikely to notice.
The backup system is already online and takes on increased load
while the main system is fixed and brought back online.
Typically, an active/active installation for a stateless service
maintains a redundant instance, and requests are load balanced using
a virtual IP address and a load balancer such as HAProxy.
A typical active/active installation for a stateful service includes
redundant services, with all instances having an identical state. In
other words, updates to one instance of a database update all other
instances. This way a request to one instance is the same as a
request to any other. A load balancer manages the traffic to these
systems, ensuring that operational systems always handle the
request.
Clusters and quorums
~~~~~~~~~~~~~~~~~~~~
The quorum specifies the minimal number of nodes
that must be functional in a cluster of redundant nodes
in order for the cluster to remain functional.
When one node fails and failover transfers control to other nodes,
the system must ensure that data and processes remain sane.
To determine this, the contents of the remaining nodes are compared
and, if there are discrepancies, a majority rules algorithm is implemented.
For this reason, each cluster in a high availability environment should
have an odd number of nodes and the quorum is defined as more than a half
of the nodes.
If multiple nodes fail so that the cluster size falls below the quorum
value, the cluster itself fails.
For example, in a seven-node cluster, the quorum should be set to
``floor(7/2) + 1 == 4``. If quorum is four and four nodes fail simultaneously,
the cluster itself would fail, whereas it would continue to function, if
no more than three nodes fail. If split to partitions of three and four nodes
respectively, the quorum of four nodes would continue to operate the majority
partition and stop or fence the minority one (depending on the
no-quorum-policy cluster configuration).
And the quorum could also have been set to three, just as a configuration
example.
.. note::
We do not recommend setting the quorum to a value less than ``floor(n/2) + 1``
as it would likely cause a split-brain in a face of network partitions.
When four nodes fail simultaneously, the cluster would continue to function as
well. But if split to partitions of three and four nodes respectively, the
quorum of three would have made both sides to attempt to fence the other and
host resources. Without fencing enabled, it would go straight to running
two copies of each resource.
This is why setting the quorum to a value less than ``floor(n/2) + 1`` is
dangerous. However it may be required for some specific cases, such as a
temporary measure at a point it is known with 100% certainty that the other
nodes are down.
When configuring an OpenStack environment for study or demonstration purposes,
it is possible to turn off the quorum checking. Production systems should
always run with quorum enabled.
Single-controller high availability mode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
OpenStack supports a single-controller high availability mode
that is managed by the services that manage highly available environments
but is not actually highly available because
no redundant controllers are configured to use for failover.
This environment can be used for study and demonstration
but is not appropriate for a production environment.
It is possible to add controllers to such an environment
to convert it into a truly highly available environment.
High availability is not for every user. It presents some challenges.
High availability may be too complex for databases or
systems with large amounts of data. Replication can slow large systems
down. Different setups have different prerequisites. Read the guidelines
for each setup.
.. important::
High availability is turned off as the default in OpenStack setups.

4161
doc/ha-guide/source/locale/en_GB/LC_MESSAGES/ha-guide.po
View File

File diff suppressed because it is too large Load Diff

3837
doc/ha-guide/source/locale/ja/LC_MESSAGES/ha-guide.po
View File

File diff suppressed because it is too large Load Diff

3971
doc/ha-guide/source/locale/tr_TR/LC_MESSAGES/ha-guide.po
View File

File diff suppressed because it is too large Load Diff

13
doc/ha-guide/source/networking-ha-dhcp.rst
View File

@ -1,13 +0,0 @@
=========================
Run Networking DHCP agent
=========================
The OpenStack Networking (neutron) service has a scheduler that lets you run
multiple agents across nodes. The DHCP agent can be natively highly available.
To configure the number of DHCP agents per network, modify the
``dhcp_agents_per_network`` parameter in the :file:`/etc/neutron/neutron.conf`
file. By default this is set to 1. To achieve high availability, assign more
than one DHCP agent per network. For more information, see
`High-availability for DHCP
<https://docs.openstack.org/newton/networking-guide/config-dhcp-ha.html>`_.

37
doc/ha-guide/source/networking-ha-l3.rst
View File

@ -1,37 +0,0 @@
=======================
Run Networking L3 agent
=======================
The Networking (neutron) service L3 agent is scalable, due to the scheduler
that supports Virtual Router Redundancy Protocol (VRRP) to distribute virtual
routers across multiple nodes. For more information about the VRRP and
keepalived, see `Linux bridge: High availability using VRRP
<https://docs.openstack.org/newton/networking-guide/config-dvr-ha-snat.html>`_
and `Open vSwitch: High availability using VRRP
<https://docs.openstack.org/newton/networking-guide/deploy-ovs-ha-vrrp.html>`_.
To enable high availability for configured routers, edit the
:file:`/etc/neutron/neutron.conf` file to set the following values:
.. tabularcolumns:: |l|l|L|
.. list-table:: /etc/neutron/neutron.conf parameters for high availability
:widths: 15 10 30
:header-rows: 1
* - Parameter
- Value
- Description
* - l3_ha
- True
- All routers are highly available by default.
* - allow_automatic_l3agent_failover
- True
- Set automatic L3 agent failover for routers
* - max_l3_agents_per_router
- 2 or more
- Maximum number of network nodes to use for the HA router.
* - min_l3_agents_per_router
- 2 or more
- Minimum number of network nodes to use for the HA router.
A new router can be created only if this number
of network nodes are available.

35
doc/ha-guide/source/networking-ha.rst
View File

@ -1,35 +0,0 @@
===================================
Configuring the networking services
===================================
.. toctree::
:maxdepth: 2
networking-ha-dhcp.rst
networking-ha-l3.rst
Configure networking on each node. See the basic information
about configuring networking in the *Networking service*
section of the
`Install Guides <https://docs.openstack.org/ocata/install/>`_,
depending on your distribution.
OpenStack network nodes contain:
- :doc:`Networking DHCP agent<networking-ha-dhcp>`
- :doc:`Neutron L3 agent<networking-ha-l3>`
- Networking L2 agent
.. note::
The L2 agent cannot be distributed and highly available.
Instead, it must be installed on each data forwarding node
to control the virtual network driver such as Open vSwitch
or Linux Bridge. One L2 agent runs per node and controls its
virtual interfaces.
.. note::
For Liberty, you can not have the standalone network nodes.
The Networking services are run on the controller nodes.
In this guide, the term `network nodes` is used for convenience.

301
doc/ha-guide/source/shared-database-configure.rst
View File

@ -1,301 +0,0 @@
==============
Configuration
==============
Before you launch Galera Cluster, you need to configure the server
and the database to operate as part of the cluster.
Configuring the server
~~~~~~~~~~~~~~~~~~~~~~~
Certain services running on the underlying operating system of your
OpenStack database may block Galera Cluster from normal operation
or prevent ``mysqld`` from achieving network connectivity with the cluster.
Firewall
---------
Galera Cluster requires that you open the following ports to network traffic:
- On ``3306``, Galera Cluster uses TCP for database client connections
and State Snapshot Transfers methods that require the client,
(that is, ``mysqldump``).
- On ``4567``, Galera Cluster uses TCP for replication traffic. Multicast
replication uses both TCP and UDP on this port.
- On ``4568``, Galera Cluster uses TCP for Incremental State Transfers.
- On ``4444``, Galera Cluster uses TCP for all other State Snapshot Transfer
methods.
.. seealso::
For more information on firewalls, see `firewalls and default ports
<https://docs.openstack.org/admin-guide/firewalls-default-ports.html>`_
in OpenStack Administrator Guide.
This can be achieved using the :command:`iptables` command:
.. code-block:: console
# iptables --append INPUT --in-interface eth0 \
--protocol tcp --match tcp --dport ${PORT} \
--source ${NODE-IP-ADDRESS} --jump ACCEPT
Make sure to save the changes once you are done. This will vary
depending on your distribution:
- For `Ubuntu <https://askubuntu.com/questions/66890/how-can-i-make-a-specific-set-of-iptables-rules-permanent#66905>`_
- For `Fedora <https://fedoraproject.org/wiki/How_to_edit_iptables_rules>`_
Alternatively, make modifications using the ``firewall-cmd`` utility for
FirewallD that is available on many Linux distributions:
.. code-block:: console
# firewall-cmd --add-service=mysql --permanent
# firewall-cmd --add-port=3306/tcp --permanent
SELinux
--------
Security-Enhanced Linux is a kernel module for improving security on Linux
operating systems. It is commonly enabled and configured by default on
Red Hat-based distributions. In the context of Galera Cluster, systems with
SELinux may block the database service, keep it from starting, or prevent it
from establishing network connections with the cluster.
To configure SELinux to permit Galera Cluster to operate, you may need
to use the ``semanage`` utility to open the ports it uses. For
example:
.. code-block:: console
# semanage port -a -t mysqld_port_t -p tcp 3306
Older versions of some distributions, which do not have an up-to-date
policy for securing Galera, may also require SELinux to be more
relaxed about database access and actions:
.. code-block:: console
# semanage permissive -a mysqld_t
.. note::
Bear in mind, leaving SELinux in permissive mode is not a good
security practice. Over the longer term, you need to develop a
security policy for Galera Cluster and then switch SELinux back
into enforcing mode.
For more information on configuring SELinux to work with
Galera Cluster, see the `SELinux Documentation
<http://galeracluster.com/documentation-webpages/selinux.html>`_
AppArmor
---------
Application Armor is a kernel module for improving security on Linux
operating systems. It is developed by Canonical and commonly used on
Ubuntu-based distributions. In the context of Galera Cluster, systems
with AppArmor may block the database service from operating normally.
To configure AppArmor to work with Galera Cluster, complete the
following steps on each cluster node:
#. Create a symbolic link for the database server in the ``disable`` directory:
.. code-block:: console
# ln -s /etc/apparmor.d/usr /etc/apparmor.d/disable/.sbin.mysqld
#. Restart AppArmor. For servers that use ``init``, run the following command:
.. code-block:: console
# service apparmor restart
For servers that use ``systemd``, run the following command:
.. code-block:: console
# systemctl restart apparmor
AppArmor now permits Galera Cluster to operate.
Database configuration
~~~~~~~~~~~~~~~~~~~~~~~
MySQL databases, including MariaDB and Percona XtraDB, manage their
configurations using a ``my.cnf`` file, which is typically located in the
``/etc`` directory. Configuration options available in these databases are
also available in Galera Cluster, with some restrictions and several
additions.
.. code-block:: ini
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
binlog_format=ROW
bind-address=10.0.0.12
# InnoDB Configuration
default_storage_engine=innodb
innodb_autoinc_lock_mode=2
innodb_flush_log_at_trx_commit=0
innodb_buffer_pool_size=122M
# Galera Cluster Configuration
wsrep_provider=/usr/lib/libgalera_smm.so
wsrep_provider_options="pc.recovery=TRUE;gcache.size=300M"
wsrep_cluster_name="my_example_cluster"
wsrep_cluster_address="gcomm://GALERA1-IP,GALERA2-IP,GALERA3-IP"
wsrep_sst_method=rsync
Configuring mysqld
-------------------
While all of the configuration parameters available to the standard MySQL,
MariaDB, or Percona XtraDB database servers are available in Galera Cluster,
there are some that you must define an outset to avoid conflict or
unexpected behavior.
- Ensure that the database server is not bound only to the localhost:
``127.0.0.1``. Also, do not bind it to ``0.0.0.0``. Binding to the localhost
or ``0.0.0.0`` makes ``mySQL`` bind to all IP addresses on the machine,
including the virtual IP address causing ``HAProxy`` not to start. Instead,
bind to the management IP address of the controller node to enable access by
other nodes through the management network:
.. code-block:: ini
bind-address=10.0.0.12
- Ensure that the binary log format is set to use row-level replication,
as opposed to statement-level replication:
.. code-block:: ini
binlog_format=ROW
Configuring InnoDB
-------------------
Galera Cluster does not support non-transactional storage engines and
requires that you use InnoDB by default. There are some additional
parameters that you must define to avoid conflicts.
- Ensure that the default storage engine is set to InnoDB:
.. code-block:: ini
default_storage_engine=InnoDB
- Ensure that the InnoDB locking mode for generating auto-increment values
is set to ``2``, which is the interleaved locking mode:
.. code-block:: ini
innodb_autoinc_lock_mode=2
Do not change this value. Other modes may cause ``INSERT`` statements
on tables with auto-increment columns to fail as well as unresolved
deadlocks that leave the system unresponsive.
- Ensure that the InnoDB log buffer is written to file once per second,
rather than on each commit, to improve performance:
.. code-block:: ini
innodb_flush_log_at_trx_commit=0
Setting this parameter to ``0`` or ``2`` can improve
performance, but it introduces certain dangers. Operating system failures can
erase the last second of transactions. While you can recover this data
from another node, if the cluster goes down at the same time
(in the event of a data center power outage), you lose this data permanently.
- Define the InnoDB memory buffer pool size. The default value is 128 MB,
but to compensate for Galera Cluster's additional memory usage, scale
your usual value back by 5%:
.. code-block:: ini
innodb_buffer_pool_size=122M
Configuring wsrep replication
------------------------------
Galera Cluster configuration parameters all have the ``wsrep_`` prefix.
You must define the following parameters for each cluster node in your
OpenStack database.
- **wsrep Provider**: The Galera Replication Plugin serves as the ``wsrep``
provider for Galera Cluster. It is installed on your system as the
``libgalera_smm.so`` file. Define the path to this file in
your ``my.cnf``:
.. code-block:: ini
wsrep_provider="/usr/lib/libgalera_smm.so"
- **Cluster Name**: Define an arbitrary name for your cluster.
.. code-block:: ini
wsrep_cluster_name="my_example_cluster"
You must use the same name on every cluster node. The connection fails
when this value does not match.
- **Cluster Address**: List the IP addresses for each cluster node.
.. code-block:: ini
wsrep_cluster_address="gcomm://192.168.1.1,192.168.1.2,192.168.1.3"
Replace the IP addresses given here with comma-separated list of each
OpenStack database in your cluster.
- **Node Name**: Define the logical name of the cluster node.
.. code-block:: ini
wsrep_node_name="Galera1"
- **Node Address**: Define the IP address of the cluster node.
.. code-block:: ini
wsrep_node_address="192.168.1.1"
Additional parameters
^^^^^^^^^^^^^^^^^^^^^^
For a complete list of the available parameters, run the
``SHOW VARIABLES`` command from within the database client:
.. code-block:: mysql
SHOW VARIABLES LIKE 'wsrep_%';
+------------------------------+-------+
| Variable_name | Value |
+------------------------------+-------+
| wsrep_auto_increment_control | ON |
+------------------------------+-------+
| wsrep_causal_reads | OFF |
+------------------------------+-------+
| wsrep_certify_nonPK | ON |
+------------------------------+-------+
| ... | ... |
+------------------------------+-------+
| wsrep_sync_wait | 0 |
+------------------------------+-------+
For documentation about these parameters, ``wsrep`` provider option, and status
variables available in Galera Cluster, see the Galera cluster `Reference
<http://galeracluster.com/documentation-webpages/reference.html>`_.

249
doc/ha-guide/source/shared-database-manage.rst
View File

@ -1,249 +0,0 @@
==========
Management
==========
When you finish installing and configuring the OpenStack database,
you can initialize the Galera Cluster.
Prerequisites
~~~~~~~~~~~~~
- Database hosts with Galera Cluster installed
- A minimum of three hosts
- No firewalls between the hosts
- SELinux and AppArmor set to permit access to ``mysqld``
- The correct path to ``libgalera_smm.so`` given to the
``wsrep_provider`` parameter
Initializing the cluster
~~~~~~~~~~~~~~~~~~~~~~~~~
In the Galera Cluster, the Primary Component is the cluster of database
servers that replicate into each other. In the event that a
cluster node loses connectivity with the Primary Component, it
defaults into a non-operational state, to avoid creating or serving
inconsistent data.
By default, cluster nodes do not start as part of a Primary Component.
In the Primary Component, replication and state transfers bring all databases
to the same state.
To start the cluster, complete the following steps:
#. Initialize the Primary Component on one cluster node. For
servers that use ``init``, run the following command:
.. code-block:: console
# service mysql start --wsrep-new-cluster
For servers that use ``systemd``, run the following command:
.. code-block:: console
# systemctl start mariadb --wsrep-new-cluster
#. Once the database server starts, check the cluster status using
the ``wsrep_cluster_size`` status variable. From the database
client, run the following command:
.. code-block:: mysql
SHOW STATUS LIKE 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 1 |
+--------------------+-------+
#. Start the database server on all other cluster nodes. For
servers that use ``init``, run the following command:
.. code-block:: console
# service mysql start
For servers that use ``systemd``, run the following command:
.. code-block:: console
# systemctl start mariadb
#. When you have all cluster nodes started, log into the database
client of any cluster node and check the ``wsrep_cluster_size``
status variable again:
.. code-block:: mysql
SHOW STATUS LIKE 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 3 |
+--------------------+-------+
When each cluster node starts, it checks the IP addresses given to
the ``wsrep_cluster_address`` parameter. It then attempts to establish
network connectivity with a database server running there. Once it
establishes a connection, it attempts to join the Primary
Component, requesting a state transfer as needed to bring itself
into sync with the cluster.
.. note::
In the event that you need to restart any cluster node, you can do
so. When the database server comes back it, it establishes
connectivity with the Primary Component and updates itself to any
changes it may have missed while down.
Restarting the cluster
-----------------------
Individual cluster nodes can stop and be restarted without issue.
When a database loses its connection or restarts, the Galera Cluster
brings it back into sync once it reestablishes connection with the
Primary Component. In the event that you need to restart the
entire cluster, identify the most advanced cluster node and
initialize the Primary Component on that node.
To find the most advanced cluster node, you need to check the
sequence numbers, or the ``seqnos``, on the last committed transaction for
each. You can find this by viewing ``grastate.dat`` file in
database directory:
.. code-block:: console
$ cat /path/to/datadir/grastate.dat
# Galera saved state
version: 3.8
uuid: 5ee99582-bb8d-11e2-b8e3-23de375c1d30
seqno: 8204503945773
Alternatively, if the database server is running, use the
``wsrep_last_committed`` status variable:
.. code-block:: mysql
SHOW STATUS LIKE 'wsrep_last_committed';
+----------------------+--------+
| Variable_name | Value |
+----------------------+--------+
| wsrep_last_committed | 409745 |
+----------------------+--------+
This value increments with each transaction, so the most advanced
node has the highest sequence number and therefore is the most up to date.
Configuration tips
~~~~~~~~~~~~~~~~~~~
Deployment strategies
----------------------
Galera can be configured using one of the following
strategies:
- Each instance has its own IP address:
OpenStack services are configured with the list of these IP
addresses so they can select one of the addresses from those
available.
- Galera runs behind HAProxy:
HAProxy load balances incoming requests and exposes just one IP
address for all the clients.
Galera synchronous replication guarantees a zero slave lag. The
failover procedure completes once HAProxy detects that the active
back end has gone down and switches to the backup one, which is
then marked as ``UP``. If no back ends are ``UP``, the failover
procedure finishes only when the Galera Cluster has been
successfully reassembled. The SLA is normally no more than 5
minutes.
- Use MySQL/Galera in active/passive mode to avoid deadlocks on
``SELECT ... FOR UPDATE`` type queries (used, for example, by nova
and neutron). This issue is discussed in the following:
- `IMPORTANT: MySQL Galera does *not* support SELECT ... FOR UPDATE
<http://lists.openstack.org/pipermail/openstack-dev/2014-May/035264.html>`_
- `Understanding reservations, concurrency, and locking in Nova
<http://www.joinfu.com/2015/01/understanding-reservations-concurrency-locking-in-nova/>`_
Configuring HAProxy
--------------------
If you use HAProxy as a load-balancing client to provide access to the
Galera Cluster, as described in the :doc:`controller-ha-haproxy`, you can
use the ``clustercheck`` utility to improve health checks.
#. Create a configuration file for ``clustercheck`` at
``/etc/sysconfig/clustercheck``:
.. code-block:: ini
MYSQL_USERNAME="clustercheck_user"
MYSQL_PASSWORD="my_clustercheck_password"
MYSQL_HOST="localhost"
MYSQL_PORT="3306"
.. note::
For Ubuntu 16.04.1: Create a configuration file for ``clustercheck``
at ``/etc/default/clustercheck``.
#. Log in to the database client and grant the ``clustercheck`` user
``PROCESS`` privileges:
.. code-block:: mysql
GRANT PROCESS ON *.* TO 'clustercheck_user'@'localhost'
IDENTIFIED BY 'my_clustercheck_password';
FLUSH PRIVILEGES;
You only need to do this on one cluster node. Galera Cluster
replicates the user to all the others.
#. Create a configuration file for the HAProxy monitor service, at
``/etc/xinetd.d/galera-monitor``:
.. code-block:: none
service galera-monitor
{
port = 9200
disable = no
socket_type = stream
protocol = tcp
wait = no
user = root
group = root
groups = yes
server = /usr/bin/clustercheck
type = UNLISTED
per_source = UNLIMITED
log_on_success =
log_on_failure = HOST
flags = REUSE
}
#. Start the ``xinetd`` daemon for ``clustercheck``. For servers
that use ``init``, run the following commands:
.. code-block:: console
# service xinetd enable
# service xinetd start
For servers that use ``systemd``, run the following commands:
.. code-block:: console
# systemctl daemon-reload
# systemctl enable xinetd
# systemctl start xinetd

32
doc/ha-guide/source/shared-database.rst
View File

@ -1,32 +0,0 @@
===============================================
Database (Galera Cluster) for high availability
===============================================
.. toctree::
:maxdepth: 2
shared-database-configure.rst
shared-database-manage.rst
The first step is to install the database that sits at the heart of the
cluster. To implement high availability, run an instance of the database on
each controller node and use Galera Cluster to provide replication between
them. Galera Cluster is a synchronous multi-master database cluster, based
on MySQL and the InnoDB storage engine. It is a high-availability service
that provides high system uptime, no data loss, and scalability for growth.
You can achieve high availability for the OpenStack database in many
different ways, depending on the type of database that you want to use.
There are three implementations of Galera Cluster available to you:
- `Galera Cluster for MySQL <http://galeracluster.com>`_: The MySQL
reference implementation from Codership, Oy.
- `MariaDB Galera Cluster <https://mariadb.org>`_: The MariaDB
implementation of Galera Cluster, which is commonly supported in
environments based on Red Hat distributions.
- `Percona XtraDB Cluster <https://www.percona.com>`_: The XtraDB
implementation of Galera Cluster from Percona.
In addition to Galera Cluster, you can also achieve high availability
through other database options, such as PostgreSQL, which has its own
replication system.

285
doc/ha-guide/source/shared-messaging.rst
View File

@ -1,285 +0,0 @@
=======================================
Messaging service for high availability
=======================================
An AMQP (Advanced Message Queuing Protocol) compliant message bus is
required for most OpenStack components in order to coordinate the
execution of jobs entered into the system.
The most popular AMQP implementation used in OpenStack installations
is RabbitMQ.
RabbitMQ nodes fail over on the application and the infrastructure layers.
The application layer is controlled by the ``oslo.messaging``
configuration options for multiple AMQP hosts. If the AMQP node fails,
the application reconnects to the next one configured within the
specified reconnect interval. The specified reconnect interval
constitutes its SLA.
On the infrastructure layer, the SLA is the time for which RabbitMQ
cluster reassembles. Several cases are possible. The Mnesia keeper
node is the master of the corresponding Pacemaker resource for
RabbitMQ. When it fails, the result is a full AMQP cluster downtime
interval. Normally, its SLA is no more than several minutes. Failure
of another node that is a slave of the corresponding Pacemaker
resource for RabbitMQ results in no AMQP cluster downtime at all.
Making the RabbitMQ service highly available involves the following steps:
- :ref:`Install RabbitMQ<rabbitmq-install>`
- :ref:`Configure RabbitMQ for HA queues<rabbitmq-configure>`
- :ref:`Configure OpenStack services to use RabbitMQ HA queues
<rabbitmq-services>`
.. note::
Access to RabbitMQ is not normally handled by HAProxy. Instead,
consumers must be supplied with the full list of hosts running
RabbitMQ with ``rabbit_hosts`` and turn on the ``rabbit_ha_queues``
option. For more information, read the `core issue
<http://people.redhat.com/jeckersb/private/vip-failover-tcp-persist.html>`_.
For more detail, read the `history and solution
<http://greenstack.die.upm.es/2015/03/02/improving-ha-failures-with-tcp-timeouts/>`_.
.. _rabbitmq-install:
Install RabbitMQ
~~~~~~~~~~~~~~~~
The commands for installing RabbitMQ are specific to the Linux distribution
you are using.
For Ubuntu or Debian:
.. code-block:: console
# apt-get install rabbitmq-server
For RHEL, Fedora, or CentOS:
.. code-block:: console
# yum install rabbitmq-server
For openSUSE:
.. code-block:: console
# zypper install rabbitmq-server
For SLES 12:
.. code-block:: console
# zypper addrepo -f obs://Cloud:OpenStack:Kilo/SLE_12 Kilo
[Verify the fingerprint of the imported GPG key. See below.]
# zypper install rabbitmq-server
.. note::
For SLES 12, the packages are signed by GPG key 893A90DAD85F9316.
You should verify the fingerprint of the imported GPG key before using it.
.. code-block:: none
Key ID: 893A90DAD85F9316
Key Name: Cloud:OpenStack OBS Project <Cloud:OpenStack@build.opensuse.org>
Key Fingerprint: 35B34E18ABC1076D66D5A86B893A90DAD85F9316
Key Created: Tue Oct 8 13:34:21 2013
Key Expires: Thu Dec 17 13:34:21 2015
For more information, see the official installation manual for the
distribution:
- `Debian and Ubuntu <https://www.rabbitmq.com/install-debian.html>`_
- `RPM based <https://www.rabbitmq.com/install-rpm.html>`_
(RHEL, Fedora, CentOS, openSUSE)
.. _rabbitmq-configure:
Configure RabbitMQ for HA queues
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. [TODO: This section should begin with a brief mention
.. about what HA queues are and why they are valuable, etc]
.. [TODO: replace "currently" with specific release names]
.. [TODO: Does this list need to be updated? Perhaps we need a table
.. that shows each component and the earliest release that allows it
.. to work with HA queues.]
The following components/services can work with HA queues:
- OpenStack Compute
- OpenStack Block Storage
- OpenStack Networking
- Telemetry
Consider that, while exchanges and bindings survive the loss of individual
nodes, queues and their messages do not because a queue and its contents
are located on one node. If we lose this node, we also lose the queue.
Mirrored queues in RabbitMQ improve the availability of service since
it is resilient to failures.
Production servers should run (at least) three RabbitMQ servers for testing
and demonstration purposes, however it is possible to run only two servers.
In this section, we configure two nodes, called ``rabbit1`` and ``rabbit2``.
To build a broker, ensure that all nodes have the same Erlang cookie file.
.. [TODO: Should the example instead use a minimum of three nodes?]
#. Stop RabbitMQ and copy the cookie from the first node to each of the
other node(s):
.. code-block:: console
# scp /var/lib/rabbitmq/.erlang.cookie root@NODE:/var/lib/rabbitmq/.erlang.cookie
#. On each target node, verify the correct owner,
group, and permissions of the file :file:`erlang.cookie`:
.. code-block:: console
# chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie
# chmod 400 /var/lib/rabbitmq/.erlang.cookie
#. Start the message queue service on all nodes and configure it to start
when the system boots. On Ubuntu, it is configured by default.
On CentOS, RHEL, openSUSE, and SLES:
.. code-block:: console
# systemctl enable rabbitmq-server.service
# systemctl start rabbitmq-server.service
#. Verify that the nodes are running:
.. code-block:: console
# rabbitmqctl cluster_status
Cluster status of node rabbit@NODE...
[{nodes,[{disc,[rabbit@NODE]}]},
{running_nodes,[rabbit@NODE]},
{partitions,[]}]
...done.
#. Run the following commands on each node except the first one:
.. code-block:: console
# rabbitmqctl stop_app
Stopping node rabbit@NODE...
...done.
# rabbitmqctl join_cluster rabbit@rabbit1
# rabbitmqctl start_app
Starting node rabbit@NODE ...
...done.
.. note::
The default node type is a disc node. In this guide, nodes
join the cluster as disc nodes. Also, nodes can join the cluster
as RAM nodes. For more details about this feature, check
`Clusters with RAM nodes <https://www.rabbitmq.com/clustering.html#ram-nodes>`_.
#. Verify the cluster status:
.. code-block:: console
# rabbitmqctl cluster_status
Cluster status of node rabbit@NODE...
[{nodes,[{disc,[rabbit@rabbit1]},{ram,[rabbit@NODE]}]}, \
{running_nodes,[rabbit@NODE,rabbit@rabbit1]}]
If the cluster is working, you can create usernames and passwords
for the queues.
#. To ensure that all queues except those with auto-generated names
are mirrored across all running nodes,
set the ``ha-mode`` policy key to all
by running the following command on one of the nodes:
.. code-block:: console
# rabbitmqctl set_policy ha-all '^(?!amq\.).*' '{"ha-mode": "all"}'
More information is available in the RabbitMQ documentation:
- `Highly Available Queues <https://www.rabbitmq.com/ha.html>`_
- `Clustering Guide <https://www.rabbitmq.com/clustering.html>`_
.. note::
As another option to make RabbitMQ highly available, RabbitMQ contains the
OCF scripts for the Pacemaker cluster resource agents since version 3.5.7.
It provides the active/active RabbitMQ cluster with mirrored queues.
For more information, see `Auto-configuration of a cluster with
a Pacemaker <https://www.rabbitmq.com/pacemaker.html>`_.
.. _rabbitmq-services:
Configure OpenStack services to use Rabbit HA queues
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Configure the OpenStack components to use at least two RabbitMQ nodes.
Use these steps to configurate all services using RabbitMQ:
#. RabbitMQ HA cluster Transport URL using ``[user:pass@]host:port`` format:
.. code-block:: ini
transport_url = rabbit://RABBIT_USER:RABBIT_PASS@rabbit1:5672,
RABBIT_USER:RABBIT_PASS@rabbit2:5672
Replace ``RABBIT_USER`` with RabbitMQ username and ``RABBIT_PASS`` with
password for respective RabbitMQ host. For more information, see
`oslo messaging transport
<https://docs.openstack.org/oslo.messaging/latest/reference/transport.html>`_.
#. Retry connecting with RabbitMQ:
.. code-block:: console
rabbit_retry_interval=1
#. How long to back-off for between retries when connecting to RabbitMQ:
.. code-block:: console
rabbit_retry_backoff=2
#. Maximum retries with trying to connect to RabbitMQ (infinite by default):
.. code-block:: console
rabbit_max_retries=0
#. Use durable queues in RabbitMQ:
.. code-block:: console
rabbit_durable_queues=true
#. Use HA queues in RabbitMQ (``x-ha-policy: all``):
.. code-block:: console
rabbit_ha_queues=true
.. note::
If you change the configuration from an old set-up
that did not use HA queues, restart the service:
.. code-block:: console
# rabbitmqctl stop_app
# rabbitmqctl reset
# rabbitmqctl start_app

12
doc/ha-guide/source/shared-services.rst
View File

@ -1,12 +0,0 @@
===============================
Configuring the shared services
===============================
This chapter describes the shared services for high availability,
such as database, messaging service.
.. toctree::
:maxdepth: 2
shared-database.rst
shared-messaging.rst

59
doc/ha-guide/source/storage-ha-backend.rst
View File

@ -1,59 +0,0 @@
.. _storage-ha-backend:
================
Storage back end
================
An OpenStack environment includes multiple data pools for the VMs:
- Ephemeral storage is allocated for an instance and is deleted when the
instance is deleted. The Compute service manages ephemeral storage and
by default, Compute stores ephemeral drives as files on local disks on the
compute node. As an alternative, you can use Ceph RBD as the storage back
end for ephemeral storage.
- Persistent storage exists outside all instances. Two types of persistent
storage are provided:
- The Block Storage service (cinder) that can use LVM or Ceph RBD as the
storage back end.
- The Image service (glance) that can use the Object Storage service (swift)
or Ceph RBD as the storage back end.
For more information about configuring storage back ends for
the different storage options, see `Manage volumes
<https://docs.openstack.org/admin-guide/blockstorage-manage-volumes.html>`_
in the OpenStack Administrator Guide.
This section discusses ways to protect against data loss in your OpenStack
environment.
RAID drives
-----------
Configuring RAID on the hard drives that implement storage protects your data
against a hard drive failure. If the node itself fails, data may be lost.
In particular, all volumes stored on an LVM node can be lost.
Ceph
----
`Ceph RBD <https://ceph.com/>`_ is an innately high availability storage back
end. It creates a storage cluster with multiple nodes that communicate with
each other to replicate and redistribute data dynamically.
A Ceph RBD storage cluster provides a single shared set of storage nodes that
can handle all classes of persistent and ephemeral data (glance, cinder, and
nova) that are required for OpenStack instances.
Ceph RBD provides object replication capabilities by storing Block Storage
volumes as Ceph RBD objects. Ceph RBD ensures that each replica of an object
is stored on a different node. This means that your volumes are protected
against hard drive and node failures, or even the failure of the data center
itself.
When Ceph RBD is used for ephemeral volumes as well as block and image storage,
it supports `live migration
<https://docs.openstack.org/admin-guide/compute-live-migration-usage.html>`_
of VMs with ephemeral drives. LVM only supports live migration of
volume-backed VMs.

189
doc/ha-guide/source/storage-ha-block.rst
View File

@ -1,189 +0,0 @@
==================================
Highly available Block Storage API
==================================
Cinder provides Block-Storage-as-a-Service suitable for performance
sensitive scenarios such as databases, expandable file systems, or
providing a server with access to raw block level storage.
Persistent block storage can survive instance termination and can also
be moved across instances like any external storage device. Cinder
also has volume snapshots capability for backing up the volumes.
Making the Block Storage API service highly available in
active/passive mode involves:
- :ref:`ha-blockstorage-pacemaker`
- :ref:`ha-blockstorage-configure`
- :ref:`ha-blockstorage-services`
In theory, you can run the Block Storage service as active/active.
However, because of sufficient concerns, we recommend running
the volume component as active/passive only.
You can read more about these concerns on the
`Red Hat Bugzilla <https://bugzilla.redhat.com/show_bug.cgi?id=1193229>`_
and there is a
`psuedo roadmap <https://etherpad.openstack.org/p/cinder-kilo-stabilisation-work>`_
for addressing them upstream.
.. _ha-blockstorage-pacemaker:
Add Block Storage API resource to Pacemaker
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On RHEL-based systems, create resources for cinder's systemd agents and create
constraints to enforce startup/shutdown ordering:
.. code-block:: console
pcs resource create openstack-cinder-api systemd:openstack-cinder-api --clone interleave=true
pcs resource create openstack-cinder-scheduler systemd:openstack-cinder-scheduler --clone interleave=true
pcs resource create openstack-cinder-volume systemd:openstack-cinder-volume
pcs constraint order start openstack-cinder-api-clone then openstack-cinder-scheduler-clone
pcs constraint colocation add openstack-cinder-scheduler-clone with openstack-cinder-api-clone
pcs constraint order start openstack-cinder-scheduler-clone then openstack-cinder-volume
pcs constraint colocation add openstack-cinder-volume with openstack-cinder-scheduler-clone
If the Block Storage service runs on the same nodes as the other services,
then it is advisable to also include:
.. code-block:: console
pcs constraint order start openstack-keystone-clone then openstack-cinder-api-clone
Alternatively, instead of using systemd agents, download and
install the OCF resource agent:
.. code-block:: console
# cd /usr/lib/ocf/resource.d/openstack
# wget https://git.openstack.org/cgit/openstack/openstack-resource-agents/plain/ocf/cinder-api
# chmod a+rx *
You can now add the Pacemaker configuration for Block Storage API resource.
Connect to the Pacemaker cluster with the :command:`crm configure` command
and add the following cluster resources:
.. code-block:: none
primitive p_cinder-api ocf:openstack:cinder-api \
params config="/etc/cinder/cinder.conf" \
os_password="secretsecret" \
os_username="admin" \
os_tenant_name="admin" \
keystone_get_token_url="http://10.0.0.11:5000/v2.0/tokens" \
op monitor interval="30s" timeout="30s"
This configuration creates ``p_cinder-api``, a resource for managing the
Block Storage API service.
The command :command:`crm configure` supports batch input, copy and paste the
lines above into your live Pacemaker configuration and then make changes as
required. For example, you may enter ``edit p_ip_cinder-api`` from the
:command:`crm configure` menu and edit the resource to match your preferred
virtual IP address.
Once completed, commit your configuration changes by entering :command:`commit`
from the :command:`crm configure` menu. Pacemaker then starts the Block Storage
API service and its dependent resources on one of your nodes.
.. _ha-blockstorage-configure:
Configure Block Storage API service
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Edit the ``/etc/cinder/cinder.conf`` file. For example, on a RHEL-based system:
.. code-block:: ini
:linenos:
[DEFAULT]
# This is the name which we should advertise ourselves as and for
# A/P installations it should be the same everywhere
host = cinder-cluster-1
# Listen on the Block Storage VIP
osapi_volume_listen = 10.0.0.11
auth_strategy = keystone
control_exchange = cinder
volume_driver = cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config = /etc/cinder/nfs_exports
nfs_sparsed_volumes = true
nfs_mount_options = v3
[database]
connection = mysql+pymysql://cinder:CINDER_DBPASS@10.0.0.11/cinder
max_retries = -1
[keystone_authtoken]
# 10.0.0.11 is the Keystone VIP
identity_uri = http://10.0.0.11:5000/
www_authenticate_uri = http://10.0.0.11:5000/
admin_tenant_name = service
admin_user = cinder
admin_password = CINDER_PASS
[oslo_messaging_rabbit]
# Explicitly list the rabbit hosts as it doesn't play well with HAProxy
rabbit_hosts = 10.0.0.12,10.0.0.13,10.0.0.14
# As a consequence, we also need HA queues
rabbit_ha_queues = True
heartbeat_timeout_threshold = 60
heartbeat_rate = 2
Replace ``CINDER_DBPASS`` with the password you chose for the Block Storage
database. Replace ``CINDER_PASS`` with the password you chose for the
``cinder`` user in the Identity service.
This example assumes that you are using NFS for the physical storage, which
will almost never be true in a production installation.
If you are using the Block Storage service OCF agent, some settings will
be filled in for you, resulting in a shorter configuration file:
.. code-block:: ini
:linenos:
# We have to use MySQL connection to store data:
connection = mysql+pymysql://cinder:CINDER_DBPASS@10.0.0.11/cinder
# Alternatively, you can switch to pymysql,
# a new Python 3 compatible library and use
# connection = mysql+pymysql://cinder:CINDER_DBPASS@10.0.0.11/cinder
# and be ready when everything moves to Python 3.
# Ref: https://wiki.openstack.org/wiki/PyMySQL_evaluation
# We bind Block Storage API to the VIP:
osapi_volume_listen = 10.0.0.11
# We send notifications to High Available RabbitMQ:
notifier_strategy = rabbit
rabbit_host = 10.0.0.11
Replace ``CINDER_DBPASS`` with the password you chose for the Block Storage
database.
.. _ha-blockstorage-services:
Configure OpenStack services to use the highly available Block Storage API
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Your OpenStack services must now point their Block Storage API configuration
to the highly available, virtual cluster IP address rather than a Block Storage
API server’s physical IP address as you would for a non-HA environment.
Create the Block Storage API endpoint with this IP.
If you are using both private and public IP addresses, create two virtual IPs
and define your endpoint. For example:
.. code-block:: console
$ openstack endpoint create volume --region $KEYSTONE_REGION \
--publicurl 'http://PUBLIC_VIP:8776/v1/%(tenant_id)s' \
--adminurl 'http://10.0.0.11:8776/v1/%(tenant_id)s' \
--internalurl 'http://10.0.0.11:8776/v1/%(tenant_id)s'

113
doc/ha-guide/source/storage-ha-file-systems.rst
View File

@ -1,113 +0,0 @@
========================================
Highly available Shared File Systems API
========================================
Making the Shared File Systems (manila) API service highly available
in active/passive mode involves:
- :ref:`ha-sharedfilesystems-pacemaker`
- :ref:`ha-sharedfilesystems-configure`
- :ref:`ha-sharedfilesystems-services`
.. _ha-sharedfilesystems-pacemaker:
Add Shared File Systems API resource to Pacemaker
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#. Download the resource agent to your system:
.. code-block:: console
# cd /usr/lib/ocf/resource.d/openstack
# wget https://git.openstack.org/cgit/openstack/openstack-resource-agents/plain/ocf/manila-api
# chmod a+rx *
#. Add the Pacemaker configuration for the Shared File Systems
API resource. Connect to the Pacemaker cluster with the following
command:
.. code-block:: console
# crm configure
.. note::
The :command:`crm configure` supports batch input. Copy and paste
the lines in the next step into your live Pacemaker configuration and then
make changes as required.
For example, you may enter ``edit p_ip_manila-api`` from the
:command:`crm configure` menu and edit the resource to match your preferred
virtual IP address.
#. Add the following cluster resources:
.. code-block:: none
primitive p_manila-api ocf:openstack:manila-api \
params config="/etc/manila/manila.conf" \
os_password="secretsecret" \
os_username="admin" \
os_tenant_name="admin" \
keystone_get_token_url="http://10.0.0.11:5000/v2.0/tokens" \
op monitor interval="30s" timeout="30s"
This configuration creates ``p_manila-api``, a resource for managing the
Shared File Systems API service.
#. Commit your configuration changes by entering the following command
from the :command:`crm configure` menu:
.. code-block:: console
# commit
Pacemaker now starts the Shared File Systems API service and its
dependent resources on one of your nodes.
.. _ha-sharedfilesystems-configure:
Configure Shared File Systems API service
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Edit the :file:`/etc/manila/manila.conf` file:
.. code-block:: ini
:linenos:
# We have to use MySQL connection to store data:
sql_connection = mysql+pymysql://manila:password@10.0.0.11/manila?charset=utf8
# We bind Shared File Systems API to the VIP:
osapi_volume_listen = 10.0.0.11
# We send notifications to High Available RabbitMQ:
notifier_strategy = rabbit
rabbit_host = 10.0.0.11
.. _ha-sharedfilesystems-services:
Configure OpenStack services to use HA Shared File Systems API
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Your OpenStack services must now point their Shared File Systems API
configuration to the highly available, virtual cluster IP address rather than
a Shared File Systems API server’s physical IP address as you would
for a non-HA environment.
You must create the Shared File Systems API endpoint with this IP.
If you are using both private and public IP addresses, you should create two
virtual IPs and define your endpoints like this:
.. code-block:: console
$ openstack endpoint create --region RegionOne \
sharev2 public 'http://PUBLIC_VIP:8786/v2/%(tenant_id)s'
$ openstack endpoint create --region RegionOne \
sharev2 internal 'http://10.0.0.11:8786/v2/%(tenant_id)s'
$ openstack endpoint create --region RegionOne \
sharev2 admin 'http://10.0.0.11:8786/v2/%(tenant_id)s'

141
doc/ha-guide/source/storage-ha-image.rst
View File

@ -1,141 +0,0 @@
==========================
Highly available Image API
==========================
The OpenStack Image service offers a service for discovering, registering, and
retrieving virtual machine images. To make the OpenStack Image API service
highly available in active/passive mode, you must:
- :ref:`glance-api-pacemaker`
- :ref:`glance-api-configure`
- :ref:`glance-services`
Prerequisites
~~~~~~~~~~~~~
Before beginning, ensure that you are familiar with the
documentation for installing the OpenStack Image API service.
See the *Image service* section in the
`Installation Guides <https://docs.openstack.org/ocata/install/>`_,
depending on your distribution.
.. _glance-api-pacemaker:
Add OpenStack Image API resource to Pacemaker
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#. Download the resource agent to your system:
.. code-block:: console
# cd /usr/lib/ocf/resource.d/openstack
# wget https://git.openstack.org/cgit/openstack/openstack-resource-agents/plain/ocf/glance-api
# chmod a+rx *
#. Add the Pacemaker configuration for the OpenStack Image API resource.
Use the following command to connect to the Pacemaker cluster:
.. code-block:: console
crm configure
.. note::
The :command:`crm configure` command supports batch input. Copy and paste
the lines in the next step into your live Pacemaker configuration and
then make changes as required.
For example, you may enter ``edit p_ip_glance-api`` from the
:command:`crm configure` menu and edit the resource to match your
preferred virtual IP address.
#. Add the following cluster resources:
.. code-block:: console
primitive p_glance-api ocf:openstack:glance-api \
params config="/etc/glance/glance-api.conf" \
os_password="secretsecret" \
os_username="admin" os_tenant_name="admin" \
os_auth_url="http://10.0.0.11:5000/v2.0/" \
op monitor interval="30s" timeout="30s"
This configuration creates ``p_glance-api``, a resource for managing the
OpenStack Image API service.
#. Commit your configuration changes by entering the following command from
the :command:`crm configure` menu:
.. code-block:: console
commit
Pacemaker then starts the OpenStack Image API service and its dependent
resources on one of your nodes.
.. _glance-api-configure:
Configure OpenStack Image service API
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Edit the :file:`/etc/glance/glance-api.conf` file
to configure the OpenStack Image service:
.. code-block:: ini
# We have to use MySQL connection to store data:
sql_connection=mysql://glance:password@10.0.0.11/glance
# Alternatively, you can switch to pymysql,
# a new Python 3 compatible library and use
# sql_connection=mysql+pymysql://glance:password@10.0.0.11/glance
# and be ready when everything moves to Python 3.
# Ref: https://wiki.openstack.org/wiki/PyMySQL_evaluation
# We bind OpenStack Image API to the VIP:
bind_host = 10.0.0.11
# Connect to OpenStack Image registry service:
registry_host = 10.0.0.11
# We send notifications to High Available RabbitMQ:
notifier_strategy = rabbit
rabbit_host = 10.0.0.11
[TODO: need more discussion of these parameters]
.. _glance-services:
Configure OpenStack services to use the highly available OpenStack Image API
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Your OpenStack services must now point their OpenStack Image API configuration
to the highly available, virtual cluster IP address instead of pointing to the
physical IP address of an OpenStack Image API server as you would in a non-HA
cluster.
For example, if your OpenStack Image API service IP address is 10.0.0.11
(as in the configuration explained here), you would use the following
configuration in your :file:`nova.conf` file:
.. code-block:: ini
[glance]
# ...
api_servers = 10.0.0.11
# ...
You must also create the OpenStack Image API endpoint with this IP address.
If you are using both private and public IP addresses, create two virtual IP
addresses and define your endpoint. For example:
.. code-block:: console
$ openstack endpoint create --region $KEYSTONE_REGION \
image public http://PUBLIC_VIP:9292
$ openstack endpoint create --region $KEYSTONE_REGION \
image admin http://10.0.0.11:9292
$ openstack endpoint create --region $KEYSTONE_REGION \
image internal http://10.0.0.11:9292

20
doc/ha-guide/source/storage-ha.rst
View File

@ -1,20 +0,0 @@
===================
Configuring storage
===================
.. toctree::
:maxdepth: 2
storage-ha-image.rst
storage-ha-block.rst
storage-ha-file-systems.rst
storage-ha-backend.rst
Making the Block Storage (cinder) API service highly available in
active/active mode involves:
* Configuring Block Storage to listen on the VIP address
* Managing the Block Storage API daemon with the Pacemaker cluster manager
* Configuring OpenStack services to use this IP address

3
tools/build-all-rst.sh
View File

@ -28,12 +28,11 @@ done
# PDF targets for Install guides are dealt in build-install-guides-rst.sh
PDF_TARGETS=( 'arch-design'\
'ha-guide' \
'image-guide' \
'install-guide')
for guide in arch-design doc-contrib-guide glossary \
ha-guide ha-guide-draft image-guide install-guide; do
ha-guide-draft image-guide install-guide; do
if [[ ${PDF_TARGETS[*]} =~ $guide ]]; then
tools/build-rst.sh doc/$guide --build build \
--target $guide $LINKCHECK $PDF_OPTION

2
www/ja/index.html
View File

@ -58,8 +58,6 @@
<a href="/mitaka/ja/networking-guide/">ネットワークガイド (Mitaka 版)</a>
<a href="/liberty/ja/networking-guide/">ネットワークガイド (Liberty 版)</a>
<p>OpenStack Networking (neutron) のデプロイと管理</p>
<a href="/ja/ha-guide/">高可用性ガイド</a>
<p>高可用性 OpenStack のためのインストールと設定方法</p>
<a href="/ja/security-guide/">セキュリティーガイド</a>
<p>より安全な OpenStack クラウド構築のためのガイドラインとシナリオ</p>
<a href="/ja/image-guide/">仮想マシンイメージガイド</a>

2
www/ocata/index.html
View File

@ -30,7 +30,7 @@
<h3 id="ops-and-admin-guides"><i class="fa fa-users"></i>Operations and Administration Guides</h3>
<a href="admin">Administrator Guides</a>
<p>Manage and troubleshoot an OpenStack cloud</p>
<a href="/ha-guide/">High Availability Guide</a><a href="/ha-guide/HAGuide.pdf">[PDF]</a>
<a href="/ha-guide/">High Availability Guide</a>
<p>Install and configure OpenStack for high availability</p>
<a href="/security-guide/">Security Guide</a>
<p>Guidelines and scenarios for creating more secure OpenStack clouds</p>

2
www/pike/index.html
View File

@ -30,7 +30,7 @@
<h3 id="ops-and-admin-guides"><i class="fa fa-users"></i>Operations and Administration Guides</h3>
<a href="admin/">Administrator Guides</a>
<p>Manage and troubleshoot an OpenStack cloud</p>
<a href="/ha-guide/">High Availability Guide</a><a href="/ha-guide/HAGuide.pdf">[PDF]</a>
<a href="/ha-guide/">High Availability Guide</a>
<p>Install and configure OpenStack for high availability</p>
<a href="/arch-design/">Architecture Guide</a>
<p>Plan and design an OpenStack cloud</p>

2
www/queens/index.html
View File

@ -30,7 +30,7 @@
<h3 id="ops-and-admin-guides"><i class="fa fa-users"></i>Operations and Administration Guides</h3>
<a href="admin/">Administrator Guides</a>
<p>Manage and troubleshoot an OpenStack cloud</p>
<a href="/ha-guide/">High Availability Guide</a><a href="/ha-guide/HAGuide.pdf">[PDF]</a>
<a href="/ha-guide/">High Availability Guide</a>
<p>Install and configure OpenStack for high availability</p>
<a href="/arch-design/">Architecture Guide</a>
<p>Plan and design an OpenStack cloud</p>

2
www/rocky/index.html
View File

@ -30,7 +30,7 @@
<h3 id="ops-and-admin-guides"><i class="fa fa-users"></i>Operations and Administration Guides</h3>
<a href="admin/">Administrator Guides</a>
<p>Manage and troubleshoot an OpenStack cloud</p>
<a href="/ha-guide/">High Availability Guide</a><a href="/ha-guide/HAGuide.pdf">[PDF]</a>
<a href="/ha-guide/">High Availability Guide</a>
<p>Install and configure OpenStack for high availability</p>
<a href="/arch-design/">Architecture Guide</a>
<p>Plan and design an OpenStack cloud</p>

414
www/static/sitemap.xml
View File

@ -41,12 +41,6 @@
<lastmod>2018-09-08T19:51:33+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/upstream-training/</loc>
@ -1013,12 +1007,6 @@
<lastmod>2018-08-15T13:18:23+0000</lastmod>
<changefreq>weekly</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/HAGuide.pdf</loc>
<lastmod>2018-09-08T19:49:42+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/image-guide/ImageGuide.pdf</loc>
@ -90725,12 +90713,6 @@
<lastmod>2018-08-30T17:22:12+0000</lastmod>
<changefreq>weekly</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/index.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/congress/queens/admin/ha-deployment.html</loc>
@ -298979,12 +298961,6 @@
<lastmod>2018-08-21T00:42:10+0000</lastmod>
<changefreq>weekly</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/</loc>
<lastmod>2018-09-08T19:53:25+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/newton/ja/install-guide-ubuntu/</loc>
@ -301361,90 +301337,6 @@
<lastmod>2017-10-11T08:24:53+0000</lastmod>
<changefreq>weekly</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/common/glossary.html</loc>
<lastmod>2018-09-08T19:53:24+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/common/app-support.html</loc>
<lastmod>2018-09-08T19:53:22+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/appendix.html</loc>
<lastmod>2018-09-08T19:53:22+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/compute-node-ha.html</loc>
<lastmod>2018-09-08T19:53:24+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/storage-ha-backend.html</loc>
<lastmod>2018-09-08T19:53:26+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/storage-ha-file-systems.html</loc>
<lastmod>2018-09-08T19:53:26+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/storage-ha-block.html</loc>
<lastmod>2018-09-08T19:53:26+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/storage-ha-image.html</loc>
<lastmod>2018-09-08T19:53:26+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/storage-ha.html</loc>
<lastmod>2018-09-08T19:53:26+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/networking-ha-l3.html</loc>
<lastmod>2018-09-08T19:53:25+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/networking-ha-dhcp.html</loc>
<lastmod>2018-09-08T19:53:25+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/networking-ha.html</loc>
<lastmod>2018-09-08T19:53:25+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/controller-ha-telemetry.html</loc>
<lastmod>2018-09-08T19:53:25+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/controller-ha-identity.html</loc>
<lastmod>2018-09-08T19:53:24+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/user-guide/cli-access-instance-through-a-console.html</loc>
@ -301607,108 +301499,12 @@
<lastmod>2017-07-18T09:01:39+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/controller-ha-memcached.html</loc>
<lastmod>2018-09-08T19:53:24+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/controller-ha-haproxy.html</loc>
<lastmod>2018-09-08T19:53:24+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/controller-ha-vip.html</loc>
<lastmod>2018-09-08T19:53:25+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/controller-ha-pacemaker.html</loc>
<lastmod>2018-09-08T19:53:25+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/intro-ha-arch-pacemaker.html</loc>
<lastmod>2018-09-08T19:53:25+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/newton/ja/install-guide-ubuntu/common/get-started-conceptual-architecture.html</loc>
<lastmod>2017-10-11T08:24:53+0000</lastmod>
<changefreq>weekly</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/controller-ha.html</loc>
<lastmod>2018-09-08T19:53:24+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/shared-messaging.html</loc>
<lastmod>2018-09-08T19:53:26+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/shared-database.html</loc>
<lastmod>2018-09-08T19:53:25+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/shared-services.html</loc>
<lastmod>2018-09-08T19:53:26+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/environment-memcached.html</loc>
<lastmod>2018-09-08T19:53:25+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/environment-ntp.html</loc>
<lastmod>2018-09-08T19:53:25+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/environment-operatingsystem.html</loc>
<lastmod>2018-09-08T19:53:25+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/environment-hardware.html</loc>
<lastmod>2018-09-08T19:53:25+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/environment.html</loc>
<lastmod>2018-09-08T19:53:25+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/intro-ha.html</loc>
<lastmod>2018-09-08T19:53:25+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/common/conventions.html</loc>
<lastmod>2018-09-08T19:53:22+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/security-guide/common/glossary.html</loc>
@ -301829,12 +301625,6 @@
<lastmod>2018-09-01T07:59:05+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/index.html</loc>
<lastmod>2018-09-08T19:53:25+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ops-guide/ops-user-facing-operations.html</loc>
@ -302081,18 +301871,6 @@
<lastmod>2017-07-19T10:36:22+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/shared-database-configure.html</loc>
<lastmod>2018-09-08T19:53:25+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ha-guide/shared-database-manage.html</loc>
<lastmod>2018-09-08T19:53:25+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ja/ops-guide/ops-maintenance-controller.html</loc>
@ -344987,120 +344765,6 @@
<lastmod>2018-09-04T06:41:21+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/networking-ha-dhcp.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/controller-ha-memcached.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/controller-ha-identity.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/networking-ha-l3.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/controller-ha-pacemaker.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/controller-ha-vip.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/compute-node-ha.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/storage-ha-file-systems.html</loc>
<lastmod>2018-09-08T19:49:29+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/controller-ha-haproxy.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/common/app-support.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/intro-ha-arch-pacemaker.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/storage-ha-backend.html</loc>
<lastmod>2018-09-08T19:49:29+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/controller-ha-telemetry.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/storage-ha.html</loc>
<lastmod>2018-09-08T19:49:29+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/appendix.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/storage-ha-block.html</loc>
<lastmod>2018-09-08T19:49:29+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/storage-ha-image.html</loc>
<lastmod>2018-09-08T19:49:29+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/networking-ha.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/common/glossary.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ko_KR/install-guide/</loc>
@ -345149,78 +344813,12 @@
<lastmod>2018-08-28T17:50:00+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/shared-database.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/environment-operatingsystem.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/environment.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/common/conventions.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/shared-messaging.html</loc>
<lastmod>2018-09-08T19:49:29+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/environment-memcached.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/intro-ha.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/environment-ntp.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/shared-services.html</loc>
<lastmod>2018-09-08T19:49:29+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/controller-ha.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/infra/manual/drivers.html</loc>
<lastmod>2018-08-28T17:50:01+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/environment-hardware.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/infra/manual/sandbox.html</loc>
@ -345353,18 +344951,6 @@
<lastmod>2015-06-12T20:49:17+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/shared-database-manage.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ha-guide/shared-database-configure.html</loc>
<lastmod>2018-09-08T19:49:28+0000</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<priority>1.0</priority>
<loc>https://docs.openstack.org/ko_KR/install-guide/firewalls-default-ports.html</loc>

2
www/stein/index.html
View File

@ -30,7 +30,7 @@
<h3 id="ops-and-admin-guides"><i class="fa fa-users"></i>Operations and Administration Guides</h3>
<a href="admin/">Administrator Guides</a>
<p>Manage and troubleshoot an OpenStack cloud</p>
<a href="/ha-guide/">High Availability Guide</a><a href="/ha-guide/HAGuide.pdf">[PDF]</a>
<a href="/ha-guide/">High Availability Guide</a>
<p>Install and configure OpenStack for high availability</p>
<a href="/arch-design/">Architecture Guide</a>
<p>Plan and design an OpenStack cloud</p>

2
www/templates/ops_and_admin_guides.tmpl
View File

@ -1,7 +1,7 @@
<h3 id="ops-and-admin-guides"><i class="fa fa-users"></i>Operations and Administration Guides</h3>
<a href="admin/">Administrator Guides</a>
<p>Manage and troubleshoot an OpenStack cloud</p>
<a href="/ha-guide/">High Availability Guide</a><a href="/ha-guide/HAGuide.pdf">[PDF]</a>
<a href="/ha-guide/">High Availability Guide</a>
<p>Install and configure OpenStack for high availability</p>
<a href="/security-guide/">Security Guide</a>
<p>Guidelines and scenarios for creating more secure OpenStack clouds</p>