This reverts commit c6259158e3eff4aff9770b7044b0179a7de533aa.
Reason for revert: cAdvisor fails with:
invalid value "percpu,referenced_memory,cpu_topology,resctrl,udp,advtcp,sched,hugetlb,memory_numa,tcp,process" for flag -disable_metrics: unsupported metric "referenced_memory" specified in disable_metrics
Change-Id: I1a0eea5c20f95f38c707401b56b7d2454484377d
Follow up fix for Ia7e923dddb77ff6db3c9160af931354a2b305e8d, which
broke the cephadm jobs.
Change-Id: Ieb39b41a6f493bd00c687610ba043a1b4e5945e7
Related-Bug: #1821696
Adds support for passing extra runtime options to cAdvisor.
By default new options disable exporting rarely useful metrics
and labels by cAdvisor. This helps reducing the load on Prometheus
and cAdvisor itself.
Change-Id: Id0144e8fa518e3236cb94ba2e3961fb455d36443
They are handled by Docker since at least 18.09 (tested).
Backport to Wallaby at most to not introduce needless restarts in
already stable branches.
Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/792583
Change-Id: Ia95355c529f1b0222dc1de06632984b6d130b9ec
With the new default since Wallaby, starting Docker makes it
enable forwarding and not filter it at all.
This may pose a security risk and should be mitigated.
Closes-Bug: #1931615
Change-Id: I5129136c066489fdfaa4d93741c22e5010b7e89d
Now that bifrost on Ubuntu focal is fully supported
both upstream (bifrost, kolla) and downstream (kayobe),
we should test it in kolla ansible CI.
Change-Id: Id3c3b7aadb9cf19813ccf3a9d39858f68526d25a
This reverts commit a967b9dd66fad5b055b071794a4b059feed9fe90.
Reason for revert: Check if it's still the case.
Change-Id: Ib5496d6bc2ddfe1fedacc4f9c859ac56fc69726e
The host list order seen during Ansible handlers may differ to the usual
play host list order, due to race conditions in notifying handlers. This
means that restart_services.yml for RabbitMQ may be included in a
different order than the rabbitmq group, resulting in a node other than
the 'first' being restarted first. This can cause some nodes to fail to
join the cluster. The include_tasks loop was introduced in [1].
This change fixes the issue by splitting the handler into two tasks, and
restarting the first node before all others.
[1] https://review.opendev.org/c/openstack/kolla-ansible/+/763137
Change-Id: I1823301d5889589bfd48326ed7de03c6061ea5ba
Closes-Bug: #1930293
Since I0474324b60a5f792ef5210ab336639edf7a8cd9e swift role uses the new
service-cert-copy role introduced in the
I6351147ddaff8b2ae629179a9bc3bae2ebac9519 but the swift role itself
doesn't contain the handler used in the service-cert-copy. Right now,
restarting the swift container isn't necessary, but the handler should
exist. Also we should fix the name of the service used.
Closes-Bug: #1931097
Change-Id: I2d0615ce6914e1f875a2647c8a95b86dd17eeb22
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
On machines with many cores, we were seeing excessive CPU load on systems
that were not very busy. With the following Erlang VM argument we saw
RabbitMQ CPU usage drop from about 150% to around 20%, on a system with
40 hyperthreads.
+S 2:2
By default RabbitMQ starts N schedulers where N is the number of CPU
cores, including hyper-threaded cores. This is fine when you assume all
your CPUs are dedicated to RabbitMQ. Its not a good idea in a typical
Kolla Ansible setup. Here we go for two scheduler threads.
More details can be found here:
https://www.rabbitmq.com/runtime.html#scheduling
and here:
https://erlang.org/doc/man/erl.html#emulator-flags
+sbwt none
This stops busy waiting of the scheduler, for more details see:
https://www.rabbitmq.com/runtime.html#busy-waiting
Newer versions of rabbit may need additional flags:
"+sbwt none +sbwtdcpu none +sbwtdio none"
But this patch should be back portable to older versions of RabbitMQ
used in Train and Stein.
Note that information on this tuning was found by looking at data from:
rabbitmq-diagnostics runtime_thread_stats
More details on that can be found here:
https://www.rabbitmq.com/runtime.html#thread-stats
Related-Bug: #1846467
Change-Id: Iced014acee7e590c10848e73feca166f48b622dc
We really want elasticsearch or monasca to catch all logs,
to providd the required centrailsed logging.
While these appears to make little material difference,
it should make it harder for logs to not get caught by
any of the outputs we have configured.
TrivialFix
Change-Id: I3bb74dcdc3cbe78cd1e1657f44e2a0af9d6508ef
monasca-thresh currently runs a local copy of the storm
to handle the threshold topology. However, it doesn't setup
the environment correctly, and the executable fails, causing
the container to continually restart.
This patch updates the container command to correctly
submit the topology to the running Apache storm. The
container will exit after it finishes the submission,
so the restart_policy is updated to on-failure, this way
if the storm is temporarily unavailable, the submission
will be retried. (NOTE: further deploys will see the
container as "changed" as it won't be running)
Patch uses KOLLA_BOOTSTRAP to trigger the container to
check if the topology is already submitted, and if so skips
the submission command so the container doesn't fail.
The config task now triggers a new reconfigure handler that
spawns a one-shot container to replace any existing topology
if the configuration has changed.
Also, all the storm.* variables in storm.yml.j2 are
removed as they were only needed for local mode and
make submitted topologies fail to load when the storm
is restarted (the referenced directories not mounted
on nimbus).
Depends-On: https://review.opendev.org/c/openstack/kolla/+/792751
Closes-Bug: #1808805
Change-Id: Ib225d76076782d695c9387e1c2693bae9a4521d7
Interface names with dashes can cause problems in Ansible since dashes
are replaced with underscores when referencing facts. In the baremetal
role we reference the fact for api_interface without replacing dashes
with underscores. This may result in host entries being omitted from
/etc/hosts.
This change fixes the issue.
Change-Id: I667adc7d8a7dbd20dbfa293f389e02355f8275bb
Related-Bug: #1927357
When using elasticsearch 7 with fluentd, you seem to get a lot
of warnings in the docker logs output that look like:
[types removal] Specifying types in bulk requests is deprecated.
The docs suggest adding suppress_type_name to stop these warnings,
and that seems to work without affecting any functionality.
Further info here:
https://github.com/uken/fluent-plugin-elasticsearch/issues/785
Closes-Bug: #1930856
Change-Id: I45be67df3717f78d78bcdc7df69600ab8681922f
Currently the logs tagged with infra.mariadb.xinetd flow into
elasticsearch with no hostname or programname attach, thus making
navigating the logs very hard.
The quick fix is renaming the tag to infra.mariadb-xinetd, which is just
enought to ensure the logs are processed correctly with the existing
filters.
TrivialFix
Change-Id: Icd72206de7c1f701bdf35c8fb3b128ef2dbe29a8
Currently when elasticsearch log output is enabled there are lots
of warnings going into elasticsearch about type being deprecated
and needing to move to @type. This change stops those warnings.
TrivialFix
Change-Id: Ideac1925cb764ad0d7d8416f56d5e4a993c6d8b6
The chrony container is deprecated in Wallaby, and disabled by default.
This change allows to remove the container if chrony is disabled.
Change-Id: I1c4436072c2d47a95625e64b731edb473384b395
