4030 Commits

Author SHA1 Message Date
Vasyl Saienko
96e9104066 [libvirt] Allow to generate dynamic config options
It may be required to use some dynamic options such as IP address
from interface where to bind service. This patch adds ability to
use dynamic logic in option detection and fill it in the configuration
file later.

Co-Authored-By: dbiletskiy <dbiletskiy@mirantis.com>

Change-Id: I8cc7da4935c11c50165a75b466d41f7d0da3e77c
2024-09-18 02:12:35 +00:00
Zuul
8a108e4bcf Merge "[libvirt] Allow to initialize virtualization modules" 2024-09-17 19:39:19 +00:00
Vasyl Saienko
f1e0981226 [helm-toolkit] Add daemonset_overrides_root util
The helm-toolkit.utils.daemonset_overrides function have some limitations:

 * it allows to override only conf values specifid in configmap-etc
 * it doesn't allow to override values for daemonsets passed via env variables
   or via damoenset definition. As result it is impossible to have mixed
   deployment when one compute is configured with dpdk while other not.
 * it is impossible to override interface names/other information stored in
   <service>-bin configmap
 * It allows to schedule on both hosts and labels, which adds some
   uncertainty

This implementation is intended to handle those limitations:

 * it allows to schedule only based on labels
 * it creates <service>-bin per daemonset override
 * it allows to override values when rendering daemonsets

 It picks data from the following structure:

  .Values:
    overrides:
      mychart_mydaemonset:
        labels:
          label::value:
            values:
              override_root_option: override_root_value
              conf:
                ovs_dpdk:
                  enabled: true
                neutron:
                  DEFAULT:
                    foo: bar

Change-Id: I5ff0f5deb34c74ca95c141f2402f375f6d926533
2024-09-17 07:43:38 +00:00
Oleksii Grudev
370f4e9f5c Remove trailing slash in endpoinds
This patch removes trailing slash in endpoint address
in case the path is empty.

Co-Authored-By: Vasyl Saienko vsaienko@mirantis.com

Change-Id: I11ace7d434b7c43f519d7ec6ac847ef94916202f
2024-09-17 06:31:35 +00:00
root
4b2d606f1d Add ability to get multiple hosts endpoint
For memcache we should set specify all hosts directly in the config
as client do key spreading based on what hosts are alive, when LB
address is used memcached can't work effectively.
This patch updates endpoint_host_lookup	to handle this scenario

Change-Id: I8c70f8e9e82bf18d04499a132ef9a016d02cea31
2024-09-17 06:30:57 +00:00
Vasyl Saienko
ef54c62fd4 Add snippet configmap_oslo_policy
Openstack policies can be applied without service restart
keep all policies in single configmap to have ability to
do not restart services on policy changes.

This patch adds a snippet of configmap that will later be used
in other helm charts.

Change-Id: I41d06df2fedb7f6cf0274c886dc9b94134507aca
2024-09-17 06:29:53 +00:00
Zuul
d97b7884da Merge "[rabbitmq] Use short rabbitmq node name" 2024-09-17 00:20:56 +00:00
Zuul
b7fd733551 Merge "[rabbitmq] Set password for guest user rabbitmq" 2024-09-17 00:20:33 +00:00
Zuul
903698565e Merge "[memcached] Allow to configure additional service parameters" 2024-09-17 00:20:32 +00:00
Zuul
87ff107a34 Merge "[mariadb] Add mariadb controller support" 2024-09-17 00:20:30 +00:00
Zuul
136c36ca33 Merge "Add service params snippet" 2024-09-17 00:20:29 +00:00
Zuul
6571a73345 Merge "[libvirt] Remove hugepages creation test" 2024-09-17 00:20:27 +00:00
Zuul
36f37b3e7f Merge "[libvirt] Handle cgroupv2 correctly" 2024-09-17 00:19:38 +00:00
Zuul
4b7a0df803 Merge "Add compute-kit-2023-1-ubuntu_focal job" 2024-09-16 21:51:43 +00:00
Zuul
d2fd9b9499 Merge "[etcd] Add cronjob with database compaction" 2024-09-16 21:51:42 +00:00
Zuul
b02dc01fa3 Merge "[etcd] Switch etcd to staetefulset" 2024-09-16 21:48:13 +00:00
Vasyl Saienko
865287258a [libvirt] Allow to initialize virtualization modules
Add init-modules libvirt container which allows to initialize
libvirt modules during start. The script is provided via
.Values.init_modules.script data structure

Change-Id: I9d5c48448b23b6b6cc18d273c9187a0a79db4af9
2024-09-16 20:09:49 +00:00
Vasyl Saienko
ea3c04a7d9 [libvirt] Remove hugepages creation test
The tests is useless as libvirt is not running in the pod
cgroup so pod settings are not applied to it.

Change-Id: Ice3957c800e29a0885a341103c453c4d6c921fd3
2024-09-16 20:09:41 +00:00
Vasyl Saienko
3903f54d0c [libvirt] Handle cgroupv2 correctly
The list of default kernel cgroup controllers may be changed
an example is kernel upgrade from 5.4.x to 5.15.x where misc controller
is enabled by default. Unhardcode list of controllers to have ability
to override them for never kernel version and allow to do not kill
qemu processes with container restart.

Change-Id: Ic4f895096a3ad2228c31f19ba1190e44f562f2a0
2024-09-16 20:09:33 +00:00
Vladimir Kozhukalov
466e2ed931 Add compute-kit-2023-1-ubuntu_focal job
This is necessary to test if libvirt changes
are compatible with cgroups v1.

Change-Id: I3cfb4e747a4cd23bc2d7051ef526fd58dc38aaf8
2024-09-16 14:59:40 -05:00
Vasyl Saienko
2f2ce5f28f [mariadb] Deploy exporter as sidecar
Deploy exporter as a sidecar to provide correct mysql metrics.

Co-Authored-By: Oleh Hryhorov <ohryhorov@mirantis.com>

Change-Id: I25cfeaf7f95f772d2b3c07a6a91220d0154b4eea
2024-09-15 09:14:53 +00:00
Vasyl Saienko
32b571ab2f [mariadb] Avoid using cluster endpoints
Switch to namespaced based endpoints to remove requirement
configure kubernetes internal cluster domain name which can't
be get from kubernetes API.

Change-Id: I8808153a83e3cec588765797d66d728bb6133a5c
2024-09-15 07:32:03 +00:00
Vasyl Saienko
10fca14997 [memcached] Allow to configure additional service parameters
Use the following structure in values to define addtional service
parameters:

Values: network:
    memcached:
      service:
        type: loadBalancer
        loadBalancerIP: 1.1.1.1
Change-Id: I94c87e530d90f603949ccacbf0602273feec741a
2024-09-14 18:08:06 +00:00
Vasyl Saienko
954e338d17 [mariadb] Add mariadb controller support
This patch adds mairadb controller that is responsible to mark one
ready pod as mariadb_role: primary to forward all traffic to it.
This will allow to drop nginx ingress controller which adds extra
hops between client and server and uses heavy customized nginx templates.

Change-Id: I3b29bc2029bfd39754516e73a09e4e14c52ccc99
2024-09-14 18:07:38 +00:00
Vasyl Saienko
d27ea24745 Add service params snippet
Allows to add custom parameters to services, and ingress services
from values as is.

Co-Authored-By: Mykyta Karpin <mkarpin@mirantis.com>

Change-Id: I42b8d07126de2cf12ddc3a934d1fd4e3a2ee0051
2024-09-14 14:59:08 +00:00
Vasyl Saienko
5b04ac3aae [etcd] Add cronjob with database compaction
etcd database need to be periodically compacted and defrag
This patch adds jobs to perform required maintenance actions
automatically.

Co-Authored-By: Oleh Hryhorov <ohryhorov@mirantis.com>

Change-Id: I31b48bb198f7322c343c7d0171322759893e374f
2024-09-14 10:33:04 +00:00
Vasyl Saienko
36288fa552 [etcd] Switch etcd to staetefulset
* Switch etcd to statefulset
* Allow to use persistant volumes to store etcd data
* Allow to deploy in clustered mode

Change-Id: I2baf5bdd05c280067991bb8b7f00c887ffd95c20
2024-09-14 09:51:00 +00:00
Vasyl Saienko
bb7580944a [rabbitmq] Use short rabbitmq node name
The patch switches rabbitmq to use short node names, this will
allow to do not care about internal domain name as it is can't
be get from k8s API.

Change-Id: I6d80bc4db4e497f7485fb5416818e0b61f821741
Related-Prod: PRODX-3456
2024-09-14 07:51:31 +00:00
Vasyl Saienko
5d086878a2 [rabbitmq] Set password for guest user rabbitmq
Guest account is enabled by default and has access to all
vhosts. Allow to change guest password during rabbitmq
configuration.

Change-Id: If23ab8d5587b13e628bce5bcb135a367324dca80
2024-09-13 15:22:31 +00:00
Vasyl Saienko
298c333ac7 [rabbitmq] Allow to bootstrap rabbitmq with initial config
Prepare rabbitmq to be running in non clustered mode, in which
it may be useful to bootstrap cluster with fresh data each time
since we do not use durable queues in openstack that are stored
on filesystem.

The two new data strucutre in rabbitmq Values are added:

  users:
    auth:
      keystone_service:
        username: keystone
        password: password
    path: /keystone
  aux_conf:
    policies:
      - vhost: "keystone"
        name: "ha_ttl_keystone"
        definition:
          ha-mode: "all"
          ha-sync-mode: "automatic"
          message-ttl: 70000
        priority: 0
        apply-to: all
        pattern: '^(?!amq\.).*'

Change-Id: Ia0dd1a8afe7b6e894bcbeafedf75131de0023df0
2024-09-13 11:32:13 +00:00
Vasyl Saienko
4b37c1fd60 [rabbitmq] Do not use hardcoded username in rabbitmq chown container
Pick up UID from .Values.pod.security_context.server.pod.runAsUser as this is
user that we are using to run service.

Change-Id: Id4c53b0a882b027e320b08ed766cb473ab9ab535
2024-09-13 10:44:55 +00:00
Vasyl Saienko
b591d3aa49 [rabbitmq] Update readiness/liveness commands
Use lightweigh rabbitmqctl ping command to check readiness and liveness probe.
check_port_connectivity - is not suatable for liveness as it does not check
  that instance of rabbitmq is actually running and we can authenticate.

Change-Id: I6f157e9aef3450dba1ad7e0cb19491a41f700bbc
2024-09-13 10:01:18 +00:00
do-gyun
61708c74d3 Decode url-encoded password for rabbit connection
Resolve that access fails when the Rabbitmq password contains special characters by the changes below.

6c5cc2fdf0

story: 2011222
task: 50999
Change-Id: I0cfc6e2228bc4b1327efb7da293849d6d1bbff19
2024-09-12 19:56:28 +00:00
Vladimir Kozhukalov
75fdad3ff9 Run utils-defragOSDs.sh in ceph-osd-default container
The Ceph defragosds cronjob script used to
connect to OSD pods not explicitly specifying
the ceph-osd-default container and eventually
tried to run the defrag script in the log-runner
container where the defrag script is mounted with
0644 permissions and shell fails to run it.

Change-Id: I4ffc6653070dbbc6f0766b278acf0ebe2b4ae1e1
2024-09-12 13:52:15 -05:00
Zuul
2163b19738 Merge "Update deploy-env role" 2024-09-05 13:20:18 +00:00
Vladimir Kozhukalov
f2bdcae040 Update deploy-env role
- Use kubeadm configuration to not set taints
  on control plain nodes (instead of removing them after
  deployment).
- Fix ssh client key permissions.
- Update the Mariadb ingress test job so it is inherinted
  from the plain compute-kit test job. And also remote
  it from the check pipeline.

Change-Id: I92c73606ed9b9161f39ea1971b3a7db7593982ff
2024-09-03 17:32:28 -05:00
astebenkova
950fc70b0c [osh-selenium] Upgrade image to ubuntu_jammy
+ run tests in a read-only file system
+ change google-chrome data directory from ~/.config/google-chrome
  (which is immutable) to /tmp/google-chrome (writable), otherwise
  Chrome fails to launch
+ activate new headless mode as the old one will be soon removed
  https://developer.chrome.com/docs/chromium/new-headless

Change-Id: I7d183b3f3d2fdc3086a5db5fa62473f777b9eb7a
2024-09-03 14:56:49 +03:00
Sergiy Markin
43fd714348 Ingress-nginx controller upgrade for mariadb
This PS bumps up ingress-nginx controller version
to v1.11.2 in mariadb chart due to CVE
vulnerability.

nginx.tmpl from mariadb chart has been updated to
match the latest 1.11.2 ingress-controller image.

Change-Id: Ie2fd811f8123515f567afde62bbbb290d58dd1b2
2024-08-27 15:36:23 +00:00
Zuul
34f945a8bc Merge "Add the ability to use custom Nagios plugins" 2024-08-23 20:43:34 +00:00
az7961
5833278b81 Add the ability to use custom Nagios plugins
Change-Id: Ib309499140994448d7b3e0eef0c875c6edb3a2ac
2024-08-23 12:59:38 -05:00
Mosher, Jaymes (jm616v)
c393d87b0d Add retry logic to index creation script
- Re-add the retry logic back to the index creation script.
- Fixed small regex bug.
- Also added function to lookup the id of a view, because the new
  views API requires an id to set the default view.
- Set noglob to make sure the asterisks in the view names aren't
  expanded.

Change-Id: Idfd56f09a739731f2ce3153b8fc284bb499a91d4
2024-08-21 16:01:46 -06:00
Zuul
15f55f32ec Merge "[ceph] Remove dependencies on legacy provisioners" 2024-08-12 15:49:33 +00:00
Stephen Taylor
3a20e59818 [ceph] Remove dependencies on legacy provisioners
The legacy RBD provisioner and the CephFS provisioner haven't been
used in some time. This change removes them.

Change-Id: I313774627fcbaed34445ebe803adf4861a0f3db5
2024-08-09 13:08:39 -06:00
okozachenko1203
af783c2fcc parse nova metadata in libvirt exporter
Change-Id: Ib49968d919bda72caffd09d57a283587ae867fec
2024-08-08 11:18:27 +10:00
Zuul
8bc357336a Merge "Updating script to use data views to support kibana 8.0 and beyond as some of api is now depreacated." 2024-08-03 18:48:57 +00:00
om9464
d9e0230365 Updating script to use data views to support kibana 8.0 and beyond
as some of api is now depreacated.

Change-Id: I58d5c388cc0f6ba56c5fe646be352a0641e0661d
2024-07-31 15:41:05 +00:00
Vladimir Kozhukalov
1178ded805 Upgrade env
- K8s 1.30.3
- Helm 3.14.0
- Crictl 1.30.1
- Calico 3.27.4
- Cilium 1.16.0
- Ingress-nginx Helm chart 4.11.1

Change-Id: I3d5a3d855b0b4b0b66e42d94e1e9704f7f91f88b
2024-07-29 05:47:12 -05:00
Vladimir Kozhukalov
05f2f45971 Add 2024.1 overrides to some charts
- Add 2024.1 overrides to those charts where
  there are overrides for previous releases.
- Update some jobs to use 2024.1 overrides.
- Update default images in  grafana, postgresql,
  nagios, ceph-rgw, ceph-provisioners,
  kubernetes-node-problem-detector
- Install tzdata package on K8s nodes. This
  is necessary for kubernetes-node-problem-detector
  chart which mounts /etc/localtime from hosts.

Change-Id: I343995c422b8d35fa902d22abf8fdd4d0f6f7334
2024-07-23 07:35:50 -05:00
Zuul
da5157ae76 Merge "Use predefined Helm repo in deployment scripts" 2024-07-19 23:00:53 +00:00
Vladimir Kozhukalov
a5f6eb6ed4 Update deploy-env role
When generating keys and sharing them between nodes
in a multinode env it is important that task which
generates keys is finished before trying to use these
keys on another node.

The PR splits the Ansible block into two blocks and
makes sure the playbook deploy-env is run with the linear
strategy. Thus we can be sure that keys are first generated
on all affected nodes and only then are used to setup
tunnels and passwordless ssh.

Change-Id: I9985855d7909aa5365876a24e2a806ab6be1dd7c
2024-07-19 12:58:39 -05:00