This reverts commit 113689ee4694de20c019735fdace447225aa18f7.
Reason for revert: systemd-252-17.el9 which includes the fix is now available in CentOS 9-stream repos.
Change-Id: I6fe19838a75a30fd5d2434c03b7f403f1c7e4b50
systemd-252-16.el9 introduced a regression
where libvirtd process exits after 120s of
inactivity.
Add a workaround to unset 120s timeout for
libvirtd, the workaround can be removed once
the fix is available in systemd rpm.
Related-Bug: #2029335
Change-Id: Id6db6c17518b54d5fef7c381c509066a569aff6d
Since we are python3 only for openstack we create a single python3
virtualenv to install all the packages into. This gives us the benefits
of installing into a virtualenv while still ensuring coinstallability.
This is a major change and will likely break many things.
There are several reasons for this. The change that started this effort
was pip stopped uninstalling packages which used distutils to generate
their package installation. Many distro packages do this which meant
that pip installed packages and distro packages could not coexist in the
global install space. More recently git has made pip installing repos as
root more difficult due to file ownership concerns.
Currently the switch to the global venv is optional, but if we go down
this path we should very quickly remove the old global installation
method as it has only caused us problems.
Major hurdles we have to get over are convincing rootwrap to trust
binaries in the virtualenvs (so you'll notice we update rootwrap
configs).
Some distros still have issues, keep them using the old setup for now.
Depends-On: https://review.opendev.org/c/openstack/grenade/+/880266
Co-Authored-By: Dr. Jens Harbott <frickler@offenerstapel.de>
Change-Id: If9bc7ba45522189d03f19b86cb681bb150ee2f25
This relaxes the limits for dbcounter to make it flush stats to the
database less often. Currently every thirty seconds or 100 hits, we
write a stats line to the database. In some services (like keystone)
this can trigger more than one write per second because of the
massive number of SELECT calls that service makes.
This removes the hit limit and decreases the mandatory flush interval
to once a minute. Hopefully this will manifest as lower load on the
database triggered by what would be readonly operations.
Change-Id: I43a58532c0541075a2d36408abc50a41f7994bda
The distutils package is deprecated and slated for removal in
Python 3.12. Let's use shutil.which which is also recomended
by PEP 632: https://peps.python.org/pep-0632/#migration-advice
Closes-Bug: #2009229
Change-Id: Ibb2a9731449e765c4a56952a9f02679e9618778b
The neutron-debug command was deprecated and finally removed,
so tools/ping_neutron.sh can no longer rely on it to create
a probe namespace. Instead, just try and use any namespace
with the network ID in it, since it's either the DHCP (ML2/OVS)
or Metadata (OVN) namespace, which should work just as well.
As this code is rarely (never?) used, this best-effort attempt
is good enough.
Change-Id: I98c992a2a774ef1fb22cee2e90ee342ab2d537ac
Depends-on: https://review.opendev.org/c/openstack/neutron/+/883081
This change enables httpd in systemd so that it
starts after a reboot and updates how selinux is
disabled to use /etc/selinux/config in addtion
to setenforce.
Change-Id: I5ea8693c0b967937483bd921b1d9984ea14bc723
We haven't been testing the distro for a while in CI, e.g. in
Tempest, the jobs on opensuse15 haven't been executed for a year
now.
Therefore the patch removes opensuse support from devstack.
Closes-Bug: #2002900
Change-Id: I0f5e4c644e2d14d1b8bb5bc0096d1469febe5fcc
Some of the API services are not properly mounted under /$service/
in the apache proxy. This patch tries to avoid recording data
for "services" like "v2.0" (in the case of neutron) by only adding
names if they're all letters. A single warning is emitted for any
services excluded by this check.
For the moment this will mean we don't collect data for those services,
but when their devstack API config is fixed, they'll start to show
up.
Change-Id: I41cc300e89a4f97a008a8ba97c91f0980f9b9c3f
We are seeing failures when using an updated setuptools version
installed together with distro pip on Ubuntu 22.04. Install the version
from u-c only when we are also installing pip from upstream.
Change-Id: Ibb6e9424e5794ccbf9a937d2eecfa3bf60ed312e
Currently, neutron tunnel endpoints must be IPv4 addresses,
i.e. $HOST_IP, although IPv6 endpoints are supported by most
drivers.
Create a TUNNEL_IP_VERSION variable to choose which host IP
to use, either HOST_IP or HOST_IPV6, and configure it in the
OVS and Linuxbridge agent driver files. The default is still
IPv4, but it can be over-ridden by specifying TUNNEL_ENDPOINT_IP
accordingly.
This behaves similar to the SERVICE_IP_VERSION option, which
can either be set to 4 or 6, but not 4+6 - the tunnel overhead
should be consistent on all systems in order not to have MTU
issues.
Must set the ML2 overlay_ip_version config option to match
else agent tunnel sync RPC will not work.
Must set the OVN external_ids:ovn-encap-ip config option to
the correct address.
Updated 'devstack-ipv6-only' job definition and verification role
that will set all services and tunnels to use IPv6 addresses.
Closes-bug: #1619476
Change-Id: I6034278dfc17b55d7863bc4db541bbdaa983a686
RHEL based distros set homedir permissions to 700,
and Ubuntu 21.04+ to 750[1], i.e missing executable
permission for group or others, this results into failures
as defined in the below bug.
Since in doc we add useradd command, it's good to
add instructions to fix the permissions there itself
instead of getting failures during installation and then
fixing it.
Also update user create script to fix permissions
by adding executable bit to DEST directory if missing.
[1] https://discourse.ubuntu.com/t/private-home-directories-for-ubuntu-21-04-onwards/19533
Closes-Bug: #1966858
Change-Id: Id2787886433281238eb95ee11a75eddeef514293
This can happen if devstack fails to run, but we still run the post
tasks. Also could happen if some sort of hybrid job configuration
does not run all of devstack but we still end up running post jobs.
Just warn to stderr and assume no DB info.
Change-Id: I211a331ab668dbb0ad7882908cca4363f865d924
There are two problems with dbcounter installation on Jammy. The first
is straightforward. We have to use `py_modules` instead of `modules` to
specify the source file. I don't know how this works on other distros
but the docs [0] seem to clearly indicate py_modules does this.
The second issue is quite an issue and requires story time. When
pip/setuptools insteall editable installs (as is done for many of the
openstack projects) it creates an easy-install.pth file that tells the
python interpreter to add the source dirs of those repos to the python
path. Normally these paths are appended to your sys.path. Pip's isolated
build env relies on the assumption that these paths are appeneded to the
path when it santizes sys.path to create the isolated environemnt.
However, when SETUPTOOLS_SYS_PATH_TECHNIQUE is set to rewrite the paths
are not appended and are inserted in the middle. This breaks pip's
isolated build env which broke dbcounter installations. We fix this by
not setting SETUPTOOLS_SYS_PATH_TECHNIQUE to rewrite. Upstream indicates
the reason we set this half a decade ago has since been fixed properly.
The reason Jammy and nothing else breaks is that python3.10 is the first
python version to use pip's isolated build envs by default.
I've locally fiddled with a patch to pip [1] to try and fix this
behavior even when rewrite is set. I don't plan to push this upstream
but it helps to illustrate where the problem lies. If someone else would
like to upstream this feel free.
Finally this change makes the jammy platform job voting again and adds
it to the gate to ensure we don't regress again.
[0] https://docs.python.org/3/distutils/sourcedist.html#specifying-the-files-to-distribute
[1] https://paste.opendev.org/show/bqVAuhgMtVtfYupZK5J6/
Change-Id: I237f5663b0f8b060f6df130de04e17e2b1695f8a
The job is broken since it is running with python3.7 and most services
now require at least python3.8.
Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: Ie21f71acffabd78c79e2b141951ccf30a5c06445
Two runs of the same job on the same patch can yield quite different
numbers for API calls if we just count the raw calls. Many of these
are tempest polling for resources, which on a slow worker can require
many more calls than a fast one.
Tempest seems to not change its User-Agent string, but the client
libraries do. So, if we ignore the regular "python-urllib" agent
calls, we get a much more stable count of service-to-service API
calls in the performance report.
Note that we were also logging in a different (less-rich) format for
the tls-proxy.log file, which hampers our ability to parse that
data in the same format. This switches it to "combined" which is used
by the access.log and contains more useful information, like the
user-agent, among other things.
Change-Id: I8889c2e53f85c41150e1245dcbe2a79bac702aad
The mysql performance_schema method for counting per-database queries
is very heavyweight in that it requires full logging (in a table) of
every query. We do hundreds of thousands in the course of a tempest
run, which ends up creating its own performance problem.
This changes the approach we take, which is to bundle a very tiny
sqlalchemy plugin module which counts just what we care about in
a special database.
It is more complex than just enabling the features in mysql, but it
is a massively smaller runtime overhead. It also provides us the
opportunity to easily zero the counters just before a tempest run.
Change-Id: I361bc30bb970cdaf18b966951f217862d302f0b9
In order to run on systems where not all requirements are present,
we should be tolerant of missing external dependencies, such as
psutil and pymysql. Print a warning (to stderr) and just leave out
those stats in that case.
Also make running the stats collector use ignore_errors:yes to avoid
failures in the future. I think the stats is not critical enough to
fail a job for bugs like this.
Related-Bug: #1970195
Change-Id: I132b0e1f5033c4f109a8b8cc776c0877574c4a49
In some cases the value is [not set], in this case
the conversion to integer does not work.
Closes-Bug: #1970431
Change-Id: I74df7d8bc9f5cbe0709a6471cf7639caea0b58e8
This makes us gather a bunch of consistent statistics after we run
tempest that can be use to measure the impact of a given change. These
are stable metrics such as "number of DB queries made" and "how much
memory is each service using after a tempest run."
Note that this will always run after devstack to generate the JSON
file, but there are two things that control its completeness:
- MYSQL_GATHER_PERFORMANCE must be enabled to get per-db stats
- Unless tls-proxy is enabled, we will only get API stats for keystone
Change-Id: Ie3b1504256dc1c9c6b59634e86fa98494bcb07b1
Since pip v22, python3.6 is not supported (the minimum version is
python3.7). This patch adds the reference for the pip3.6 URL to be
used instead of the default one.
Closes-Bug: #1959600
Change-Id: Iab2c391d5388461fe9e9037cee81884ce8032e72
Running get-pip.py fails on Ubuntu when running twice, e.g. after a
unstack/stack cycle. Just use distro pip instead.
Closes-Bug: #1957048
Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: I87a8d53ed8860dd017a6c826dee6b6f4baef3c96
Recent iputils release in CentOS 8-stream causing
ping failures with non root user. This needs a fix
in systemd package as mentioned in the Related Bugs,
until it's fixed and is in 8-stream mirrors let's
workaround it by setting net.ipv4.ping_group_range
setting manually.
Related-Bug: #1957941
Related-Bug: rhbz#2037807
Change-Id: I0d8dac910647968b625020c2a94e626ba5255058
openEuler is an open-source Linux based operating system. The current
openEuler kernel is based on Linux and supports multi arch, such as X86_64
and aarch64. It fully unleashes the potential of computing chips. As an
efficient, stable, and secure open-source OS built by global open-source
contributors, openEuler applies to database, big data, cloud computing,
and AI scenarios. openEuler is using RPM for package management.
Note:
Currently there is no available package for uwsgi-plugin-python3 and ovn, so that
openEuler needs manually install them from source.
Website: https://www.openeuler.org/en/
Change-Id: I169a0017998054604a63ac6c177d0f43f8a32ba6
Co-Authored-By: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: Kevin Zhao <kevin.zhao@linaro.org>
This patch includes changes required to run devstack on CentOS Stream 9
which has been already published in official repos by CentOS team [1]:
- Add RDO deps repository for CS9.
- remove xinetd package from installation for swift. Note that
rsync-daemon is installed which should work fine.
- Replace genisoimage by xorriso in CS9.
- Use /etc/os-release to identify the distro in CS9 as it doesn't
provide lsb_release command.
- Use pip from rpm package instead of from get-pip.py as done in Fedora.
- Add non-voting job devstack-platform-centos-9-stream to the check
pipeline.
Change-Id: Ic67cddabd5069211dc0611994b8b8360bcd61bef
Keystone no longer has any special functionality hidden behind the admin
site. KEYSTONE_AUTH_URI which used to point to the admin site has long
ago been changed to be a copy of KEYSTONE_SERVICE_URI, which points to
the public site.
Drop all KEYSTONE_AUTH_* variables except KEYSTONE_AUTH_URI which may
still be in use in some plugins.
This also allows to finally drop the fixup_keystone() function.
Change-Id: I549f3cadc27d137e014241cdd47e90267859c848
The use of this function has been deprecated for a long time[0]. With
PyYAML==6.0 the call is now failing, so replace it with the safe
version.
[0] https://msg.pyyaml.org/load
Signed-off-by: Jens Harbott <frickler@offenerstapel.de>
Change-Id: I7a170262b50a5c80a516095b872d52e1bea5479d
The official Ubuntu cloud images have some further python pkgs
preinstalled that conflict with our requirements. Allow to
overwrite them.
Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Closes-Bug: 1871485
Change-Id: I793c250cae5e7b9bc835b7016d790d1f9ae8a7f3
continue is not used in a proper context here (outside of loop). Use
null cmd instead to simply fall through the pip installation.
Signed-off-by: Michal Berger <michallinuxstuff@gmail.com>
Change-Id: Iaea2e5c0177b475edf19d08d71933a74debbb5d9
The uninstall here has been around since
Ibb4b42119dc2e51577c77bbbbffb110863e5324d. At the time, there might
have been conflicts between packaged and installed pip. We don't need
it today; get-pip.py keeps itself separate enough in /usr/local on all
platforms. Thus we can also remove the suse/centos special-casing.
python3-pip is in the RPM list so we don't need to
re-install for Fedora.
Add a note on why we are over-installing pip.
Remove some old setuptools workarounds that are commented out.
Change-Id: Ie3cb81a8ff71cf4b81e23831c380f83b0381de71
