40 Commits

Author SHA1 Message Date
Michal Nasiadka
4bc410c6ca haproxy: support single external frontend
Use case: exposing single external https frontend and
load balancing services using FQDNs.

Support different ports for internal and external endpoints.

Introduced kolla_url filter to normalize urls like:
- https://magnum.external:443/v1
- http://magnum.external:80/v1

Change-Id: I9fb03fe1cebce5c7198d523e015280c69f139cd0
Co-Authored-By: Jakub Darmach <jakub@stackhpc.com>
2023-06-29 01:44:00 +02:00
Matt Crees
6c2aace8d6 Integrate oslo-config-validator
Regularly, we experience issues in Kolla Ansible deployments because we
use wrong options in OpenStack configuration files. This is because
OpenStack services ignore unknown options. We also need to keep on top
of deprecated options that may be removed in the future. Integrating
oslo-config-validator into Kolla Ansible will greatly help.

Adds a shared role to run oslo-config-validator on each service. Takes
into account that services have multiple containers, and these may also
use multiple config files. Service roles are extended to use this shared
role. Executed with the new command ``kolla-ansible validate-config``.

Change-Id: Ic10b410fc115646d96d2ce39d9618e7c46cb3fbc
2022-12-21 17:19:09 +00:00
Michal Nasiadka
1aac65de0c Fix issues introduced by ansible-lint 6.6.0
mainly jinja spacing and jinja[invalid] related

Change-Id: I6f52f2b0c1ef76de626657d79486d31e0f47f384
2022-09-21 14:34:54 +00:00
Zuul
89c3a92066 Merge "Add api_workers for each service to defaults" 2022-08-22 15:30:33 +00:00
Michal Arbet
baad47ac61 Edit services roles to support database sharding
Depends-On: https://review.opendev.org/c/openstack/kolla/+/769385
Depends-On: https://review.opendev.org/c/openstack/kolla/+/765781

Change-Id: I3c4182a6556dafd2c936eaab109a068674058fca
2022-08-09 12:15:26 +02:00
Michal Arbet
3e8db91a1e Add api_workers for each service to defaults
Render {{ openstack_service_workers }} for workers
of each openstack service is not enough. There are
several services which has to have more workers because
there are more requests sent to them.

This patch is just adding default value for workers for
each service and sets {{ openstack_service_workers }} as
default, so value can be overrided in hostvars per server.
Nothing changed for normal user.

Change-Id: Ifa5863f8ec865bbf8e39c9b2add42c92abe40616
2022-07-12 20:09:16 +02:00
Radosław Piliszek
3e75a33ad4 Use the new image naming scheme
Change-Id: Ib4b15ed4feac82d8492b1c0f0238a752eac668e6
2022-05-23 06:37:25 +00:00
Marcin Juszkiewicz
1620ab5be9 drop install_type from image names
We have only one value for install_type now and it gets removed from
image names.

Change-Id: I8bf95fd7aa9dd26b80d618ca0fcb097003b4cb0a
2022-04-20 12:29:12 +02:00
Pierre Riteau
56fc74f231 Move project_name and kolla_role_name to role vars
Role vars have a higher precedence than role defaults. This allows to
import default vars from another role via vars_files without overriding
project_name (see related bug for details).

Change-Id: I3d919736e53d6f3e1a70d1267cf42c8d2c0ad221
Related-Bug: #1951785
2021-12-31 09:26:25 +00:00
Dr. Jens Harbott
479a78706a Stop creating non-keystone admin endpoints
The admin interface for endpoints never had any real use, the
functionality was the same as for the public or internal endpoints,
except for Keystone. Even for Keystone with API v3 it would no longer
really be needed, but it is still being required by some libraries that
cannot be changed in order to stay backwards compatible.

Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: Icf3bf08deab2c445361f0a0124d87ad8b0e4e9d9
2021-12-21 13:09:36 +01:00
Zuul
44c7b166cf Merge "Use Docker healthchecks for mistral services" 2021-09-08 16:07:51 +00:00
Mark Goddard
ade5bfa302 Use ansible_facts to reference facts
By default, Ansible injects a variable for every fact, prefixed with
ansible_. This can result in a large number of variables for each host,
which at scale can incur a performance penalty. Ansible provides a
configuration option [0] that can be set to False to prevent this
injection of facts. In this case, facts should be referenced via
ansible_facts.<fact>.

This change updates all references to Ansible facts within Kolla Ansible
from using individual fact variables to using the items in the
ansible_facts dictionary. This allows users to disable fact variable
injection in their Ansible configuration, which may provide some
performance improvement.

This change disables fact variable injection in the ansible
configuration used in CI, to catch any attempts to use the injected
variables.

[0] https://docs.ansible.com/ansible/latest/reference_appendices/config.html#inject-facts-as-vars

Change-Id: I7e9d5c9b8b9164d4aee3abb4e37c8f28d98ff5d1
Partially-Implements: blueprint performance-improvements
2021-06-23 10:38:06 +01:00
LinPeiWen
561be6de32 Use Docker healthchecks for mistral services
This change enables the use of Docker healthchecks for mistral services.
Implements: blueprint container-health-check

Change-Id: I5c60d22936d0e2e92fc9e6bcbf0a869d3f0b1687
2021-03-26 14:28:36 +00:00
Rafael Weingärtner
f425c0678f Standardize use and construction of endpoint URLs
The goal for this push request is to normalize the construction and use
 of internal, external, and admin URLs. While extending Kolla-ansible
 to enable a more flexible method to manage external URLs, we noticed
 that the same URL was constructed multiple times in different parts
 of the code. This can make it difficult for people that want to work
 with these URLs and create inconsistencies in a large code base with
 time. Therefore, we are proposing here the use of
 "single Kolla-ansible variable" per endpoint URL, which facilitates
 for people that are interested in overriding/extending these URLs.

As an example, we extended Kolla-ansible to facilitate the "override"
of public (external) URLs with the following standard
"<component/serviceName>.<companyBaseUrl>".
Therefore, the "NAT/redirect" in the SSL termination system (HAproxy,
HTTPD or some other) is done via the service name, and not by the port.
This allows operators to easily and automatically create more friendly
 URL names. To develop this feature, we first applied this patch that
 we are sending now to the community. We did that to reduce the surface
  of changes in Kolla-ansible.

Another example is the integration of Kolla-ansible and Consul, which
we also implemented internally, and also requires URLs changes.
Therefore, this PR is essential to reduce code duplicity, and to
facility users/developers to work/customize the services URLs.

Change-Id: I73d483e01476e779a5155b2e18dd5ea25f514e93
Signed-off-by: Rafael Weingärtner <rafael@apache.org>
2020-08-19 07:22:17 +00:00
Mark Goddard
146b00efa7 Mount /etc/timezone based on host OS
Previously we mounted /etc/timezone if the kolla_base_distro is debian
or ubuntu. This would fail prechecks if debian or ubuntu images were
deployed on CentOS. While this is not a supported combination, for
correctness we should fix the condition to reference the host OS rather
than the container OS, since that is where the /etc/timezone file is
located.

Change-Id: Ifc252ae793e6974356fcdca810b373f362d24ba5
Closes-Bug: #1882553
2020-08-10 10:14:18 +01:00
Dincer Celik
4b5df0d866 Introduce /etc/timezone to Debian/Ubuntu containers
Some services look for /etc/timezone on Debian/Ubuntu, so we should
introduce it to the containers.

In addition, added prechecks for /etc/localtime and /etc/timezone.

Closes-Bug: #1821592
Change-Id: I9fef14643d1bcc7eee9547eb87fa1fb436d8a6b3
2020-04-09 18:53:36 +00:00
Mark Goddard
5a786436be Python 3: Use distro_python_version for dev mode
In dev mode currently the python source is mounted under python2.7
site-packages. This change fixes this to use the distro_python_version
variable to ensure dev mode works with Python 3 images.

Change-Id: Ieae3778a02f1b79023b4f1c20eff27b37f481077
Partially-Implements: blueprint python-3
2020-01-30 14:00:34 +00:00
Mark Goddard
9755c924be CentOS 8: Support variable image tag suffix
For the CentOS 7 to 8 transition, we will have a period where both
CentOS 7 and 8 images are available. We differentiate these images via a
tag - the CentOS 8 images will have a tag of train-centos8 (or
master-centos8 temporarily).

To achieve this, and maintain backwards compatibility for the
openstack_release variable, we introduce a new 'openstack_tag' variable.
This variable is based on openstack_release, but has a suffix of
'openstack_tag_suffix', which is empty except on CentOS 8 where it has a
value of '-centos8'.

Change-Id: I12ce4661afb3c255136cdc1aabe7cbd25560d625
Partially-Implements: blueprint centos-rhel-8
2020-01-10 09:56:04 +00:00
Radosław Piliszek
bc053c09c1 Implement IPv6 support in the control plane
Introduce kolla_address filter.
Introduce put_address_in_context filter.

Add AF config to vars.

Address contexts:
- raw (default): <ADDR>
- memcache: inet6:[<ADDR>]
- url: [<ADDR>]

Other changes:

globals.yml - mention just IP in comment

prechecks/port_checks (api_intf) - kolla_address handles validation

3x interface conditional (swift configs: replication/storage)

2x interface variable definition with hostname
(haproxy listens; api intf)

1x interface variable definition with hostname with bifrost exclusion
(baremetal pre-install /etc/hosts; api intf)

neutron's ml2 'overlay_ip_version' set to 6 for IPv6 on tunnel network

basic multinode source CI job for IPv6

prechecks for rabbitmq and qdrouterd use proper NSS database now

MariaDB Galera Cluster WSREP SST mariabackup workaround
(socat and IPv6)

Ceph naming workaround in CI
TODO: probably needs documenting

RabbitMQ IPv6-only proto_dist

Ceph ms switch to IPv6 mode

Remove neutron-server ml2_type_vxlan/vxlan_group setting
as it is not used (let's avoid any confusion)
and could break setups without proper multicast routing
if it started working (also IPv4-only)

haproxy upgrade checks for slaves based on ipv6 addresses

TODO:

ovs-dpdk grabs ipv4 network address (w/ prefix len / submask)
not supported, invalid by default because neutron_external has no address
No idea whether ovs-dpdk works at all atm.

ml2 for xenapi
Xen is not supported too well.
This would require working with XenAPI facts.

rp_filter setting
This would require meddling with ip6tables (there is no sysctl param).
By default nothing is dropped.
Unlikely we really need it.

ironic dnsmasq is configured IPv4-only
dnsmasq needs DHCPv6 options and testing in vivo.

KNOWN ISSUES (beyond us):

One cannot use IPv6 address to reference the image for docker like we
currently do, see: https://github.com/moby/moby/issues/39033
(docker_registry; docker API 400 - invalid reference format)
workaround: use hostname/FQDN

RabbitMQ may fail to bind to IPv6 if hostname resolves also to IPv4.
This is due to old RabbitMQ versions available in images.
IPv4 is preferred by default and may fail in the IPv6-only scenario.
This should be no problem in real life as IPv6-only is indeed IPv6-only.
Also, when new RabbitMQ (3.7.16/3.8+) makes it into images, this will
no longer be relevant as we supply all the necessary config.
See: https://github.com/rabbitmq/rabbitmq-server/pull/1982

For reliable runs, at least Ansible 2.8 is required (2.8.5 confirmed
to work well). Older Ansible versions are known to miss IPv6 addresses
in interface facts. This may affect redeploys, reconfigures and
upgrades which run after VIP address is assigned.
See: https://github.com/ansible/ansible/issues/63227

Bifrost Train does not support IPv6 deployments.
See: https://storyboard.openstack.org/#!/story/2006689

Change-Id: Ia34e6916ea4f99e9522cd2ddde03a0a4776f7e2c
Implements: blueprint ipv6-control-plane
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-10-16 10:24:35 +02:00
Mark Goddard
3522d235bd Refactor service, endpoint and user registration
Use upstream Ansible modules for registration of services, endpoints,
users, projects, roles, and role grants.

Change-Id: I7c9138d422cc91c177fd8992347176bb54156b5a
2019-09-17 10:13:56 -07:00
Rafael Weingärtner
22a6223b1b Standardize the configuration of "oslo_messaging" section
After all of the discussions we had on
"https://review.opendev.org/#/c/670626/2", I studied all projects that
have an "oslo_messaging" section. Afterwards, I applied the same method
that is already used in "oslo_messaging" section in Nova, Cinder, and
others. This guarantees that we have a consistent method to
enable/disable notifications across projects based on components (e.g.
Ceilometer) being enabled or disabled. Here follows the list of
components, and the respective changes I did.

* Aodh:
The section is declared, but it is not used. Therefore, it will
be removed in an upcomming PR.

* Congress:
The section is declared, but it is not used. Therefore, it will
be removed in an upcomming PR.

* Cinder:
It was already properly configured.

* Octavia:
The section is declared, but it is not used. Therefore, it will
be removed in an upcomming PR.

* Heat:
It was already using a similar scheme; I just modified it a little bit
to be the same as we have in all other components

* Ceilometer:
Ceilometer publishes some messages in the rabbitMQ. However, the
default driver is "messagingv2", and not ''(empty) as defined in Oslo;
these configurations are defined in ceilometer/publisher/messaging.py.
Therefore, we do not need to do anything for the
"oslo_messaging_notifications" section in Ceilometer

* Tacker:
It was already using a similar scheme; I just modified it a little bit
to be the same as we have in all other components

* Neutron:
It was already properly configured.

* Nova
It was already properly configured. However, we found another issue
with its configuration. Kolla-ansible does not configure nova
notifications as it should. If 'searchlight' is not installed (enabled)
the 'notification_format' should be 'unversioned'. The default is
'both'; so nova will send a notification to the queue
versioned_notifications; but that queue has no consumer when
'searchlight' is disabled. In our case, the queue got 511k messages.
The huge amount of "stuck" messages made the Rabbitmq cluster
unstable.

https://bugzilla.redhat.com/show_bug.cgi?id=1478274
https://bugs.launchpad.net/ceilometer/+bug/1665449

* Nova_hyperv:
I added the same configurations as in Nova project.

* Vitrage
It was already using a similar scheme; I just modified it a little bit
to be the same as we have in all other components

* Searchlight
I created a mechanism similar to what we have in AODH, Cinder, Nova,
and others.

* Ironic
I created a mechanism similar to what we have in AODH, Cinder, Nova,
and others.

* Glance
It was already properly configured.

* Trove
It was already using a similar scheme; I just modified it a little bit
to be the same as we have in all other components

* Blazar
It was already using a similar scheme; I just modified it a little bit
to be the same as we have in all other components

* Sahara
It was already using a similar scheme; I just modified it a little bit
to be the same as we have in all other components

* Watcher
I created a mechanism similar to what we have in AODH, Cinder, Nova,
and others.

* Barbican
I created a mechanism similar to what we have in Cinder, Nova,
and others. I also added a configuration to 'keystone_notifications'
section. Barbican needs its own queue to capture events from Keystone.
Otherwise, it has an impact on Ceilometer and other systems that are
connected to the "notifications" default queue.

* Keystone
Keystone is the system that triggered this work with the discussions
that followed on https://review.opendev.org/#/c/670626/2. After a long
discussion, we agreed to apply the same approach that we have in Nova,
Cinder and other systems in Keystone. That is what we did. Moreover, we
introduce a new topic "barbican_notifications" when barbican is
enabled. We also removed the "variable" enable_cadf_notifications, as
it is obsolete, and the default in Keystone is CADF.

* Mistral:
It was hardcoded "noop" as the driver. However, that does not seem a
good practice. Instead, I applied the same standard of using the driver
and pushing to "notifications" queue if Ceilometer is enabled.

* Cyborg:
I created a mechanism similar to what we have in AODH, Cinder, Nova,
and others.

* Murano
It was already using a similar scheme; I just modified it a little bit
to be the same as we have in all other components

* Senlin
It was already using a similar scheme; I just modified it a little bit
to be the same as we have in all other components

* Manila
It was already using a similar scheme; I just modified it a little bit
to be the same as we have in all other components

* Zun
The section is declared, but it is not used. Therefore, it will
be removed in an upcomming PR.

* Designate
It was already using a similar scheme; I just modified it a little bit
to be the same as we have in all other components

* Magnum
It was already using a similar scheme; I just modified it a little bit
to be the same as we have in all other components

Closes-Bug: #1838985

Change-Id: I88bdb004814f37c81c9a9c4e5e491fac69f6f202
Signed-off-by: Rafael Weingärtner <rafael@apache.org>
2019-08-15 13:18:16 -03:00
binhong.hua
12ff28a693 Make kolla-ansible support extra volumes
When integrating 3rd party component into openstack with kolla-ansible,
maybe have to mount some extra volumes to container.

Change-Id: I69108209320edad4c4ffa37dabadff62d7340939
Implements: blueprint support-extra-volumes
2019-05-17 11:55:04 +08:00
Eduardo Gonzalez
db4e25836d Deploy mistral-event-engine service
This PS adds deployment of mistral-event-engine service
within mistral role.

Change-Id: If187e1a49a321d1e85fb3cfff2ea037e567b30dc
2018-11-06 17:50:54 +00:00
Adam Harwell
f1c8136556 Refactor haproxy config (split by service) V2.0
Having all services in one giant haproxy file makes altering
configuration for a service both painful and dangerous. Each service
should be configured with a simple set of variables and rendered with a
single unified template.

Available are two new templates:

* haproxy_single_service_listen.cfg.j2: close to the original style, but
only one service per file
* haproxy_single_service_split.cfg.j2: using the newer haproxy syntax
for separated frontend and backend

For now the default will be the single listen block, for ease of
transition.

Change-Id: I6e237438fbc0aa3c89a3c8bd706a53b74e71904b
2018-09-26 03:30:38 -07:00
MinSun
12f4554330 Support checkout dedicated version from git with dev mode
Now kolla dev mode only support clone master branch from git,
add version tag to support clone dedicated branch.

Change-Id: I88de238e5dc7461ba0662a3ecea9a2d80fd0db60
2018-08-14 16:06:00 +08:00
Lakshmi Prasanna Goutham Pratapa
14bf524756 Apply Resource Constraints to Services.
This commit is to apply resource-constraints to a few more OpenStack services.
Commit to  apply constraints to the last set of services will be made in
the upcoming commit.

Depends-on: Icafa54baca24d2de64238222a5677b9d8b90e2aa
Change-Id: I39004f54281f97d53dfa4b1dbcf248650ad6f186
2018-07-26 11:35:28 +00:00
caoyuan
c3c7b4827e dev mode: Add support for mistral
Provide support for kolla dev mode in mistral. When
'kolla_dev_mode' or 'mistral_dev_mode' variables are
enabled, source code of mistral project is cloned
and bind mounted.

Partially implements: blueprint mount-sources
Co-Authored-By: chenqiaomin <chen.qiaomin@99cloud.net>

Change-Id: I6e5f9b1a6b35dbb672905987e715cae4bf10af7f
2018-05-12 22:20:41 +08:00
Alexandru Bogdan Pica
8e3b79440c Implement external MariaDB and pre-configured Databases support
This change allows the following use cases:

1. Using an already-configured MariaDB / MySQL server / Cluster
2. Using already-created DB users, without requiring root DB access.

Update: added external mariadb precheck

Change-Id: I78b0d178306d7c5293b0bf53e445f19f18b4b824
Implements: blueprint external-mariadb-support.
Closes-Bug: #1603121
2018-01-23 13:07:40 +00:00
caoyuan
b63ecd7e3e Update kolla_internal_fqdn to database_address since it defined in all.yml
through the database_address has beed defined in groups_vars/all.yml, we should
better use it, this way, if we want to use external database, we just need to
redefined in all.yml

refer to https://github.com/openstack/kolla-ansible/blob/master/ansible/group_vars/all.yml#L83
Co-Authored-By: chenqiaomin <chen.qiaomin@99cloud.net>

Change-Id: Ie559301451954e16347ceaabf02f594c5c5cbe56
2018-01-17 13:01:48 +08:00
Dai Dang Van
cda878d2fb Mixing binary and source images for M* projects
In case Kolla's users want to deploy with both of
binary and source image, we should have a variable
install type that define install type for each project.

We also add specific image tag for each Openstack project.

This commit is implemented for Magnum, Manila,
Mistral and Murano projects.

Change-Id: I6aaeb74f791c79229430a52b066327de7219a9b1
Implements: blueprint mixing-binary-and-source-image
2017-06-03 15:12:49 +07:00
Jenkins
8884422f3d Merge "Refactor register.yml files" 2017-01-26 18:48:59 +00:00
Jeffrey Zhang
8155d74d8d Refactor register.yml files
Co-Authored-By: Mauricio Lima <mauriciolimab@gmail.com>
Change-Id: I9a4a6b6523dee4b388513386b7d85d421f2b7b89
2017-01-26 17:10:27 +00:00
caoyuan
7148b818e0 Optimize reconfiguration for mistral
Change-Id: Ieec58fe3759c8ce430a187357ca8282b0f81b7eb
Partially-implements: blueprint better-reconfigure
2017-01-24 13:52:31 +00:00
Eduardo Gonzalez
a71874f24f Fix mistral service
- Add mistral in HAproxy
- Set mistral api to bind on api_interface
- Fix mistral endpoint
- Add database population on bootstraping
- Add mistral port prechecks

Change-Id: If1617fb9dcd8b3bbd4f94c68ca87c36e39711016
Closes-Bug: #1626570
2016-09-28 17:21:51 +02:00
Waldemar Znoinski
a2c2eba34f add support for non-default database_port in kolla/ansible
This change makes each step of the kolla deployment aware
of the port database was configured to listen on.
It defaults mariadb_port to database_port.

Change-Id: I8e85d5732015afc0a5481cb33e0b629fdfa84a1b
Closes-Bug: #1576151
DocImpact
2016-05-13 12:12:25 +00:00
SamYaple
57124620ab Fix Keystone v3 and Horizon
After our switch to keystone-manage bootstrap Horizon is not happy
due to v3 not being setup correctly. This patch fixes that

This also includes removal of unused variables (transforms them into
endpoint url variables)

TrivialFix
Change-Id: I1e04db8c24049f80e974c063f03068a2ab32a563
2016-03-03 15:44:53 +00:00
SamYaple
d3cfb2052a Change kolla_internal_address variable
Due to poor planning on our variable names we have a situation where
we have "internal_address" which must be a VIP, but "external_address"
which should be a DNS name. Now with two vips "external_vip_address"
is a new variable.

This corrects that issue by deprecating kolla_internal_address and
replacing it with 4 nicely named variables.

kolla_internal_vip_address
kolla_internal_fqdn
kolla_external_vip_address
kolla_external_fqdn

The default behaviour will remain the same, and the way the variable
inheritance is setup the kolla_internal_address variable can still be
set in globals.yml and propogate out to these 4 new variables like it
normally would, but all reference to kolla_internal_address has been
completely removed.

Change-Id: I4556dcdbf4d91a8d2751981ef9c64bad44a719e5
Partially-Implements: blueprint ssl-kolla
2016-02-26 20:00:09 +00:00
venkatamahesh
70de590a11 Use uppercase 'S' in word "OpenStack"
Change-Id: I27ca0ea75f3d6a4371c91b3cb2c7a999ec90fbc4
2016-02-15 12:37:30 +00:00
Ice Yao
e1ff26612f remove option verbose define in defaults/main.yml
References: https://review.openstack.org/#/c/269042/

TrivialFix

Change-Id: Ief08781342a06f956fc4cf00ba4383759da8c897
2016-01-20 16:17:05 +08:00
hparekh
409fc11e36 Ansible files for mistral containers.
Change-Id: Ifb55ff4a7ca70bc3d10fbb2ec067c245342d3288
Partially-implements: blueprint enable-mistral-container
2016-01-05 09:40:14 +09:00