10432 Commits

Author SHA1 Message Date
Clark Boylan
1d5be95196 Cleanup comment that should've been removed
The previous change, I237f5663b0f8b060f6df130de04e17e2b1695f8a, removed
a SETUPTOOLS flag, but not the comment explaining why that flag was
previously set. Clean up that comment.

Change-Id: I32b0240fd56310d7f10596aaa8ef432679bfd66a
2022-05-23 08:46:50 -07:00
Clark Boylan
50e3c06ec2 Fix dbcounter installation on Jammy
There are two problems with dbcounter installation on Jammy. The first
is straightforward. We have to use `py_modules` instead of `modules` to
specify the source file. I don't know how this works on other distros
but the docs [0] seem to clearly indicate py_modules does this.

The second issue is quite an issue and requires story time. When
pip/setuptools insteall editable installs (as is done for many of the
openstack projects) it creates an easy-install.pth file that tells the
python interpreter to add the source dirs of those repos to the python
path. Normally these paths are appended to your sys.path. Pip's isolated
build env relies on the assumption that these paths are appeneded to the
path when it santizes sys.path to create the isolated environemnt.

However, when SETUPTOOLS_SYS_PATH_TECHNIQUE is set to rewrite the paths
are not appended and are inserted in the middle. This breaks pip's
isolated build env which broke dbcounter installations. We fix this by
not setting SETUPTOOLS_SYS_PATH_TECHNIQUE to rewrite. Upstream indicates
the reason we set this half a decade ago has since been fixed properly.

The reason Jammy and nothing else breaks is that python3.10 is the first
python version to use pip's isolated build envs by default.

I've locally fiddled with a patch to pip [1] to try and fix this
behavior even when rewrite is set. I don't plan to push this upstream
but it helps to illustrate where the problem lies. If someone else would
like to upstream this feel free.

Finally this change makes the jammy platform job voting again and adds
it to the gate to ensure we don't regress again.

[0] https://docs.python.org/3/distutils/sourcedist.html#specifying-the-files-to-distribute
[1] https://paste.opendev.org/show/bqVAuhgMtVtfYupZK5J6/

Change-Id: I237f5663b0f8b060f6df130de04e17e2b1695f8a
2022-05-20 10:35:18 -07:00
Zuul
34c2842676 Merge "Configure placement section in neutron conf" 2022-05-19 17:23:50 +00:00
Dr. Jens Harbott
083eeee5af Make jammy platform jobs non-voting
We missed to add the jobs to the gate queue and so they have already
regressed before they were actually in place. Make them non-voting for
now until the issues are fixed.

Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: I5d1f83dfe23747096163076dcf80750585c0260e
2022-05-19 13:55:35 +02:00
Zuul
08254ca312 Merge "Add Ubuntu 22.04 LTS (jammy) platform job" 2022-05-19 08:16:18 +00:00
Zuul
2f889954ce Merge "lib/tempest: add wait for Glance image import" 2022-05-18 15:38:49 +00:00
Brian Rosmaita
111a38b4d6 lib/tempest: add wait for Glance image import
Glance image import is asynchronous and may be configured to do image
conversion.  If image import is being used, it's possible that the
tempest configuration code is executed before the import has
completed and there may be no active images yet.  In that case,
we will poll glance every TEMPEST_GLANCE_IMPORT_POLL_INTERVAL seconds
(default: 1) to see if there are TEMPEST_GLANCE_IMAGE_COUNT active
images (default: 1) up to TEMPEST_GLANCE_IMPORT_POLL_LIMIT times
(default: 12).

You can see an example of the issue this patch addresses in real
life:
https://review.opendev.org/c/openstack/glance/+/841278/1#message-456096e48b28e5b866deb8bf53e9258ee08219a0

Change-Id: Ie99f12691d9062611a8930accfa14d9540970cc5
2022-05-18 08:22:49 -04:00
Zuul
9eb64896dd Merge "Use proper sed separator for paths" 2022-05-18 11:30:49 +00:00
92a34dbe95 Configure placement section in neutron conf
Without it segment plugin fails to connect with
placement api. Configure the placement section
if service is deployed.

Closes-Bug: #1973783
Change-Id: Ie7f37770a04f622735cf2263c601257669ab5064
2022-05-18 15:27:40 +05:30
Zuul
47b7b84422 Merge "Improve API log parsing" 2022-05-18 00:19:23 +00:00
Zuul
4fd2831753 Merge "Change DB counting mechanism" 2022-05-18 00:19:20 +00:00
Zuul
071374fa05 Merge "Collect status of all services" 2022-05-17 18:40:28 +00:00
Zuul
83a81755ac Merge "Wait for OVN dbs also along with sockets" 2022-05-12 19:03:49 +00:00
Dan Smith
64d68679d9 Improve API log parsing
Two runs of the same job on the same patch can yield quite different
numbers for API calls if we just count the raw calls. Many of these
are tempest polling for resources, which on a slow worker can require
many more calls than a fast one.

Tempest seems to not change its User-Agent string, but the client
libraries do. So, if we ignore the regular "python-urllib" agent
calls, we get a much more stable count of service-to-service API
calls in the performance report.

Note that we were also logging in a different (less-rich) format for
the tls-proxy.log file, which hampers our ability to parse that
data in the same format. This switches it to "combined" which is used
by the access.log and contains more useful information, like the
user-agent, among other things.

Change-Id: I8889c2e53f85c41150e1245dcbe2a79bac702aad
2022-05-12 07:55:30 -07:00
Dan Smith
fe52d7f0a8 Change DB counting mechanism
The mysql performance_schema method for counting per-database queries
is very heavyweight in that it requires full logging (in a table) of
every query. We do hundreds of thousands in the course of a tempest
run, which ends up creating its own performance problem.

This changes the approach we take, which is to bundle a very tiny
sqlalchemy plugin module which counts just what we care about in
a special database.

It is more complex than just enabling the features in mysql, but it
is a massively smaller runtime overhead. It also provides us the
opportunity to easily zero the counters just before a tempest run.

Change-Id: I361bc30bb970cdaf18b966951f217862d302f0b9
2022-05-12 07:55:02 -07:00
Dr. Jens Harbott
5c765cb8a1 Add Ubuntu 22.04 LTS (jammy) platform job
The new Ubuntu LTS release has been made last week, start running
devstack on it as a platform job.

Horizon has issues with py310, so gets disabled for now.

Run variants with OVS and OVN(default).

Co-Authored-By: yatinkarel <ykarel@redhat.com>
Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: I47696273d6b009f754335b44ef3356b4f5115cd8
2022-05-12 13:54:02 +02:00
Zuul
d450e146cc Merge "Global option for enforcing scope (ENFORCE_SCOPE)" 2022-05-07 10:51:35 +00:00
1baa8905d5 Wait for OVN dbs also along with sockets
When OVN is setup from distro packages, the
main service is ovn-central which when restarted,
restarts ovn-northd, ovn nb and db services.

And during the restart ovn dbs(ovnnb_db.db and ovnsb_db.db)
are created, which may sometime takes time as seen with
ubuntu jammy tests[1].

We already checking for socket's file to be available,
let's also check for db files as without it ovn-*ctl
operations succeed but changes are not persisted until
db files are available and changes are lost with the restart.

[1] https://review.opendev.org/c/openstack/devstack/+/839389

Change-Id: I178da7af8cba8bcc8a67174e439df7c0f2c7d4d5
2022-05-06 18:06:18 +05:30
42be2425d8 Collect status of all services
Would be helpful in troubleshooting services
which either fails to start or takes time to
start.

Related-Bug: #1970679
Change-Id: Iba2fce5f8b1cd00708f092e6eb5a1fbd96e97da0
2022-04-29 16:40:32 +00:00
Zuul
85c2999e27 Merge "Tolerate missing deps in get-stats.py" 2022-04-27 22:49:07 +00:00
Zuul
bfae1bee79 Merge "Set public bridge up for v6 only configurations" 2022-04-27 10:21:07 +00:00
Zuul
48417ca241 Merge "Drop centos 8 stream testing" 2022-04-26 19:57:12 +00:00
Dan Smith
1b601c7b1e Tolerate missing deps in get-stats.py
In order to run on systems where not all requirements are present,
we should be tolerant of missing external dependencies, such as
psutil and pymysql. Print a warning (to stderr) and just leave out
those stats in that case.

Also make running the stats collector use ignore_errors:yes to avoid
failures in the future. I think the stats is not critical enough to
fail a job for bugs like this.

Related-Bug: #1970195
Change-Id: I132b0e1f5033c4f109a8b8cc776c0877574c4a49
2022-04-26 08:02:39 -07:00
Harald Jensås
bab0c92103 Use tryint() for stats value
In some cases the value is [not set], in this case
the conversion to integer does not work.

Closes-Bug: #1970431
Change-Id: I74df7d8bc9f5cbe0709a6471cf7639caea0b58e8
2022-04-26 15:51:35 +02:00
Julia Kreger
6964ba4a98 Set public bridge up for v6 only configurations
A long time ago, Ironic's IPv6 only job started to fail working with
errors indicated the host was unreacable. Turns out, this was because
the $ext_gw_interface was not being set to up, and thus could
be found in a Down state, and thus the kernel would not accept routes
for it.

Adds an explicit step to turn up the public bridge, much as done in
the IPv4 router plugin code which would also be executed in 4+6.

That being said, Ironic's CI jobs are very intentionally IPv6 only
to ensure that we have no chances of v4 addressing getting used
at any point in time.

This should allow Ironic to return it's IPv6 only CI job back
to the normal check queue, once a ironic plugin issue has been
resolved which was introduced while it was removed.

Change-Id: I121ec8a2e9640b21a7126f2eeb23da36b4aa95bf
2022-04-26 06:37:31 -07:00
Grzegorz Grasza
8615563df4 Global option for enforcing scope (ENFORCE_SCOPE)
This updates each devstack service library, to use it as the
default value for service-specific RBAC configuration.

Change-Id: I41061d042206c411ee3dd94ce91098e612af7ae7
2022-04-26 14:17:20 +02:00
Ghanshyam Mann
c6dfd169ae Drop centos 8 stream testing
In Zed cycle testing runtime, we are targetting the centos 9 stream
- https://governance.openstack.org/tc/reference/runtimes/zed.html

With dropping the python 3.6 support, project started adding python 3.8
as minimum, example nova:
- 56b5aed08c/setup.cfg (L13)

with that, centos 8 stream job is failing 100%
- https://zuul.openstack.org/build/970d029dc96742c3aa0f6932a35e97cf
- https://zuul.openstack.org/builds?job_name=devstack-platform-centos-8-stream&skip=0

This commit drops centos-8-stream testing so that we focus on centos-9-stream.

Change-Id: I045e67b1ca79aba1b2a7be9f88d7804c69c6d781
2022-04-25 15:24:39 -05:00
Zuul
3b0c035b90 Merge "install mod_ssl on centos 9 stream by default" 2022-04-25 20:04:16 +00:00
Balazs Gibizer
7191c5e7e7 Use proper sed separator for paths
I941ef5ea90970a0901236afe81c551aaf24ac1d8 added a sed command that
should match and delete path values but used '/' as sed separator. This
leads to error in unstack.sh runs when the path also contains '/':

+./unstack.sh:main:188 sudo sed -i '/directory=/opt/stack/ d' /etc/gitconfig
sed: -e expression #1, char 13: unknown command: `o'

So this patch replace '/' separator with '+'.

Change-Id: I06811c0d9ee7ecddf84ef1c6dd6cff5129dbf4b1
2022-04-25 15:26:28 +02:00
Zuul
76c519bde6 Merge "modify the sample value of LOGDAYS" 2022-04-22 11:09:36 +00:00
Zuul
45f71b10ef Merge "Gather performance data after tempest" 2022-04-21 23:06:35 +00:00
Sean Mooney
af75f689fa install mod_ssl on centos 9 stream by default
This change adds mod_ssl to the default set of rpms installed
on rpm based distros.

this is required if the tls-proxy service is enabled
for multi node centos based jobs.

Change-Id: I52652de88352094c824da68e5baf7db4c17cb027
2022-04-21 20:40:44 +01:00
Zhou Yanbing
4423450eb3 modify the sample value of LOGDAYS
the value of LOGDAYS in samples/local.conf is 2, so change the
value in the comment and the sample value in the document to
be consistent with it.

Change-Id: I5822bbf1d6ad347c67c886be1e3325113d079114
2022-04-21 15:00:41 +08:00
Dan Smith
c2772c2984 Gather performance data after tempest
This makes us gather a bunch of consistent statistics after we run
tempest that can be use to measure the impact of a given change. These
are stable metrics such as "number of DB queries made" and "how much
memory is each service using after a tempest run."

Note that this will always run after devstack to generate the JSON
file, but there are two things that control its completeness:

 - MYSQL_GATHER_PERFORMANCE must be enabled to get per-db stats
 - Unless tls-proxy is enabled, we will only get API stats for keystone

Change-Id: Ie3b1504256dc1c9c6b59634e86fa98494bcb07b1
2022-04-20 13:07:22 -07:00
Zuul
d380858b2d Merge "Add OpenStack two nodes nodeset for Centos 9" 2022-04-19 05:48:02 +00:00
Dan Smith
4baeb3b51f Write safe.directory items to system git config
This is necessary for more consistent behavior across multiple
distro versions. Apparently somewhere along the way, git started
looking at the current user's home directory instead of $HOME.

Related-Bug: https://bugs.launchpad.net/devstack/+bug/1968798

Change-Id: I941ef5ea90970a0901236afe81c551aaf24ac1d8
2022-04-18 08:05:44 -07:00
Ian Wienand
676dcaf944 Mark our source trees as safe for git to use as other users
git commit [1] introduced a new behaviour to work around a CVE that
disallows any git operations in directories not owned by the current
user.

This may seem unrelated to installation, but it plays havoc with PBR,
which calls out to git to get to get revision history.  So if you are
"pip install"-ing from a source tree you don't own, the PBR git calls
in that tree now fail and the install blows up.

This plays havoc with our model.  Firstly, we checkout all code as
"stack" then install it globally with "sudo" (i.e. root) -- which
breaks.  We also have cases of essentially the opposite -- checkouts
we have installed as root, but then run tox in them as a regular user;
tox wants to install the source in its venv but now we have another
user conflict.

This uses the only available configuration option to avoid that by
globally setting the source directories we clone as safe.  This is an
encroachment of the global system for sure, but is about the only
switch available at the moment.  For discussion of other approaches,
see [2].

Related-Bug: https://bugs.launchpad.net/devstack/+bug/1968798

[1] 8959555cee
[2] https://review.opendev.org/c/openstack/devstack/+/837636

Change-Id: Ib9896a99b6d6c4d359ee412743ce30512b3c4fb7
2022-04-13 16:49:07 +10:00
Zuul
ce0ff1fd9d Merge "ensure /usr/local/bin in in path" 2022-04-11 14:57:13 +00:00
Sean Mooney
eca9783a0a ensure /usr/local/bin in in path
osc is typicaly installed in /usr/local/bin
to avoid command not found errors when invoking osc
in devstack ensure that /usr/local/bin is included
in the PATH.

Change-Id: I605fbc4b131149bf5d1b6307b360fe365c680b1a
2022-04-07 13:31:50 +01:00
afariasa
f4a703661e Add OpenStack two nodes nodeset for Centos 9
Change-Id: I01c8e5e0e88d0dcfe778f19548a2e268406ef6bf
2022-04-06 15:24:09 +00:00
Zuul
aac6b6c791 Merge "Drop setup.py and setup.cfg" 2022-03-29 17:50:05 +00:00
Zuul
729b196445 Merge "Move openEuler job to experimental pipeline" 2022-03-29 14:03:58 +00:00
Ghanshyam Mann
45b029064f Move openEuler job to experimental pipeline
OpenEuler job fails 100% of the time. As discussed in QA meeting,
we agreed to move OpenEuler job to experimental pipeline.
- https://meetings.opendev.org/meetings/qa/2022/qa.2022-03-22-15.00.log.html#l-76

Once it is fixed, we can think of adding back to regular pipeline.

Change-Id: I831889a09fabe5bed5522d17e352ec8009eac321
2022-03-29 07:18:40 +00:00
Zuul
0ed70e3f76 Merge "Update DEVSTACK_SERIES to zed" 2022-03-28 13:02:03 +00:00
Dr. Jens Harbott
5c51a95d10 Drop setup.py and setup.cfg
devstack isn't a python project, these were introduced only for docs
building and made redundant with [0]. We can remove them now.

[0] Iedcc008b170821aa74acefc02ec6a243a0dc307c

Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: I90ca1c6918c016d10c579fbae49d13fff1ed59af
2022-03-28 14:00:54 +02:00
Zuul
e79913c65f Merge "ignore failures to copy the devstack cache" 2022-03-27 09:57:23 +00:00
zhouyanbing
8dc342d400 remove unuseful local variable define
the local varibale: api_cell_conf in start_nova_rest function
is unuseful, so remove it now.

Change-Id: I0019ce807cf3905ee246b684fce2abcb46336306
2022-03-26 14:22:23 +08:00
Martin Kopec
189c7ff142 Update DEVSTACK_SERIES to zed
stable/yoga branch has been created now and
current master is for zed.

Change-Id: I8743a3440a0ce96acb24b34971548b43ae7c8d4c
2022-03-25 14:06:52 +01:00
Zuul
14779fc992 Merge "Clean usage of project_id in the Neutron's L3 service module" 2022-03-24 03:21:17 +00:00
Slawek Kaplonski
cebd00aa04 Clean usage of project_id in the Neutron's L3 service module
After patch [1] project_id in that module is no longer needed as to make
it working with new secure RBAC policies we had to hardcode "demo"
project to be used always.
This is small follow-up patch with cleaning after [1].

[1] https://review.opendev.org/c/openstack/devstack/+/826851/

Change-Id: Iddf9692817c91807fc3269547910e4f83585f07f
2022-03-23 15:30:38 +01:00