7217 Commits

Author SHA1 Message Date
Abhishek Kekane
1d8117b093 Refresh Glance example configs for Xena milestone 2
Change-Id: I034497da6cd977ad6d025bf183f51c3ff1d3df40
2021-07-09 05:06:40 +00:00
Zuul
77be0e3e23 Merge "Revert "Remove all usage of keystoneclient"" 2021-07-06 17:11:49 +00:00
Zuul
8c8e1a94f8 Merge "Fix oslo policy DeprecatedRule warnings" 2021-07-06 11:03:55 +00:00
Ghanshyam Mann
2dc1084fcd Fix oslo policy DeprecatedRule warnings
Since 3.7.0, oslo policy started the DeprecationWarning[1] if
deprecated_reason and deprecated_since param are not passed
in DeprecatedRule or they are passed in RuleDefault object.

These warnings are logged for every test which increase the
log size and sometime can full the log buffer and fail the
job.
- https://zuul.opendev.org/t/openstack/build/ddb517bc792b49b9a8cf508eb3e361b0/log/job-output.txt#918

[1] https://github.com/openstack/oslo.policy/blob/3.7.0/oslo_policy/policy.py#L1538

Change-Id: Iae5d4c06c736e9cb24037f32cbe369fdfee9c2ab
2021-07-05 11:48:17 -05:00
Erno Kuvaja
2d28173752 Fix the policy deprecation message
Removed the defaulting to roles part of the deprecation message.

Change-Id: I37d67457efdd8572d41cd839fb00c84f1f197892
Closes-Bug: #1934673
2021-07-05 14:41:19 +01:00
Dan Smith
da771fce25 Add unified quotas documentation
Related to blueprint glance-unified-quotas

Change-Id: I59d0b48f0652120eaae39509685345a12447933e
2021-07-02 08:29:07 -07:00
Dan Smith
8d6ee6f822 Add image_count_uploading quota enforcement
This makes us enforce a quota on the total number of upload-related
image operations owned by a user.

Partially-implements: blueprint glance-unified-quotas

Change-Id: I2a28750aaf968e6a6324eb194d4280a640bfa5aa
2021-07-02 08:29:07 -07:00
Dan Smith
084a77e644 Add user_get_uploading_count() to DB API
This adds an efficient query method for getting a count of the user's
images that are in any state of uploading. This includes images
in the process of saving from upload, staging, importing, or being
copied to another store.

Partially-implements: blueprint glance-unified-quotas

Change-Id: Iebba15771000e86a0701d20ed9620eb578c2cb85
2021-07-02 08:29:06 -07:00
Zuul
91ed096ec0 Merge "Add a nonvoting functional job with RBAC defaults" 2021-07-01 15:19:36 +00:00
Zuul
e788d68ef4 Merge "Use default policies in our tests" 2021-07-01 14:35:50 +00:00
Dan Smith
8201f5f101 Add a nonvoting functional job with RBAC defaults
This adds a new tox target called functional-py38-rbac which enables
the new secure RBAC policy defaults for all functional tests. To do
this, the functional tests needed a little bit of extra work to
actually set those, and a new non-voting job is added to run these
in CI.

Related to blueprint policy-refactor

Change-Id: Id376193521671bdb0ebc08ea8e563578bbaa541f
2021-06-30 11:49:30 -07:00
Abhishek Kekane
21257615e7 Use default policies in our tests
Made changes to use default policies in our unit and functional tests
rather than referring it from glance/tests/etc/policy.yaml file. Existing
function 'set_policy_rules' can be used to test custom policy rules in
functional testing.

Improvements needed in followup patch:
1. Property protection related unit tests still reads the special policy
from the policy.yaml file, need to make provision to override it instead.

2. Need to remove 'set_policy' function which actually responsible for
above behavior.

Related to blueprint policy-refactor
Change-Id: I0de9b9f9a7de499574101e3366ced730b0cb5fd1
2021-06-30 16:12:25 +00:00
Dan Smith
a36666e2fe Add image_count_total quota enforcement
This makes us enforce a quota on the total number of (non-deleted)
images owned by a user.

Partially-implements: blueprint glance-unified-quotas

Change-Id: I8af124d9307263cd8289d0701fb9a745d13b1d56
2021-06-29 08:53:18 -07:00
Dan Smith
d3d6a646e3 Add user_get_image_count() to DB API
This adds an efficient query method for getting a count of a user's
non-deleted images. It can be used by subsequent patches to enforce
the image count quota.

Partially-implements: blueprint glance-unified-quotas

Change-Id: I4d6e071b8d48862032f36c80ee19632a3fe85009
2021-06-29 08:53:18 -07:00
Dan Smith
59990d513a Add image_stage_total quota enforcement
This makes us enforce a quota on the amount of data a user has in
staging.

Partially-implements: blueprint glance-unified-quotas

Change-Id: I3cca4e589adc0aec138e5933c311aefd69ccee51
2021-06-29 08:53:18 -07:00
Dan Smith
5261fab3ea Add user_get_staging_usage() to DB API
This adds a method that counts the sizes of images that are in the
process of doing an import, indicating that they are using staging
storage resources. We gather images that are in obvious states of
being created via import, as well as any images that appear to be
active but in the middle of a copy, since those could be using staging
resources as well.

This is somewhat mirrored from user_get_storage_usage(). Since there
was no test for that method, this adds a test case for both that and
the new method.

Partially-implements: blueprint glance-unified-quotas

Change-Id: Ic2b045549ac0d6112b0c565d2e993576a5626799
2021-06-29 08:53:18 -07:00
Dan Smith
76c3011a64 Enforce keystone limits for image upload
This adds enforcement of the image_size_total keystone limit for
image upload and import. We simply check the quota before either of
these operations and refuse to proceed further if the user is over
their quota.

Note that this disables checking of the global size quota if keystone
quotas are enabled.

Note this includes another fix to couple unit tests that do not
properly pass context to the get_flow() method.

Partially-implements: blueprint glance-unified-quotas
Change-Id: Idf5f004b72436df1f9c77bb32d60b9be5ae77a68
2021-06-29 08:53:18 -07:00
Zuul
f3eb5214a3 Merge "Add unified quotas infrastructure" 2021-06-28 18:57:07 +00:00
Zuul
7b0eb314cc Merge "Drop lower-constraints jobs" 2021-06-28 18:56:51 +00:00
Dan Smith
7c1cd438a0 Fix broken test_update_queued_image_with_hidden
This fixes the test to behave the way we expect. It was failing to
do the update because it was using an image the requester did not
own, and asserting the found behavior of 403. However, the intent
here is to allow it to be updated. So, this uses the proper image and
asserts the proper behavior.

Change-Id: I71afe6a877485c8f92e67dcf32bb475c1a1a42a3
Closes-Bug: #1933360
2021-06-25 06:54:33 +00:00
Zuul
11d0355d55 Merge "Make image stage set image.size" 2021-06-24 17:27:41 +00:00
Zuul
2e1eff0fbb Merge "Update image.size after conversion" 2021-06-24 09:34:04 +00:00
Zuul
3c6356eea7 Merge "Refactor SynchronousAPIBase for more cases" 2021-06-24 03:04:44 +00:00
Zuul
36ab1f5203 Merge "Changed minversion in tox to 3.18.0" 2021-06-23 20:58:48 +00:00
Zuul
35ab03ab3e Merge "Remove references to sys.version_info" 2021-06-23 20:41:31 +00:00
Zuul
f1fb383067 Merge "Replace collections.Iterable" 2021-06-23 20:38:55 +00:00
Dan Smith
06e6542f15 Add unified quotas infrastructure
This adds some infrastructure to be able to query and honor limits
declared in keystone. It adds a single initial quota value for the
total size of all active images for bootstrapping the tests.

Checking these values is controlled by a new configuration option
that globally enables and disables the checking, defaulting to
False.

Related to blueprint glance-unified-quotas
Change-Id: I8d8f4aaed465486e80be85bc9a5d2c2be7f1ecad
2021-06-21 10:58:55 -07:00
Dan Smith
fa1bc2f904 Drop lower-constraints jobs
Change-Id: Id030b56b721cb62255936d56f0409c4be9ce21d4
2021-06-21 07:19:01 -07:00
Dan Smith
c290c4af5a Make image stage set image.size
When we stage an image, we know the image size, so we should set it.
Since the user may be streaming an image and did not declare the
expected size, this lets them confirm before they import. It also
provides us a value to count for the staging quota coming in a later
patch.

The bulk of this change is adjusting existing tests to validate the
size for all of our stage/import tests.

This follows the change to set the image size during conversion,
ensuring that we can set it during stage for non-conversion jobs,
and that the conversion code that re-sets it after changing the
image data and size continues to work.

Related to blueprint glance-unified-quotas

Change-Id: I93a9145df27594a0cc59828619a7d0573e58d4fc
2021-06-21 07:19:01 -07:00
Dan Smith
154ef3fe94 Update image.size after conversion
When we convert an image to a specified format during import, we
update the disk_format to match. At that point, we also know the (new)
image.size, so we should set it.

This is somewhat related to setting image size on stage, in that once
it is set we will validate that it does not change in later steps.
Since this one comes between stage and the actual store upload, this
patch makes conversion set it and confirms that the later steps are
happy with that. A later patch sets it during stage, confirming that
we can change it here during conversion when we are changing the
actual image file itself.

Related to blueprint glance-unified-quotas

Change-Id: I795c52f606f85955e39efc29b75f2941be1264b4
2021-06-21 07:19:01 -07:00
Dan Smith
a1e46cca6e Refactor SynchronousAPIBase for more cases
This simply adds some functionality and flexibility to this test base
so that subsequent patches can do some more things.

Related to blueprint glance-unified-quotas

Change-Id: Ic9359aac5dba2b4d7d0d2c7fa92a5b67440e22e0
2021-06-21 07:19:01 -07:00
Zuul
049a8cd447 Merge "Allow member creation when using db.simple api" 2021-06-14 15:41:55 +00:00
Erno Kuvaja
9b683678b2 Revert "Remove all usage of keystoneclient"
This reverts commit 810417df865c7b4eb71cee4dfee843d65972852b.

The Swift driver not being able to use Trusts had nothing to do
about the store driver itself nor that keystoneauth1 would have
broken the feature, but rather it not having the functionality
in the first place and us not catching that on reviews.

We should figure out how to test this before we try to replace
this code again.

Change-Id: If12a013404296486dc387b099477d1608b24ba63
Closes-Bug: #1916052
2021-06-10 13:51:49 +01:00
Dan Smith
4f20e5007f Fix missing context args to get_flow()
TaskExecutor._get_flow() passes a context argument to the task
initialization routine, but a couple of tests that short-circuit that
do not. This makes those tests mimic the real behavior separately
from the later patch that needs it to make it clear that it is not
actually requiring a new argument at runtime.

Related to blueprint glance-unified-quotas

Change-Id: Ifc2adcb7f8eaa7da2e7b063a8b79175025582500
2021-06-04 07:11:14 -07:00
wu.shiming
bba5014c1a Changed minversion in tox to 3.18.0
The patch bumps min version of tox to 3.18.0 in order to
replace tox's whitelist_externals by allowlist_externals option:
https://github.com/tox-dev/tox/blob/master/docs/changelog.rst#v3180-2020-07-23

Change-Id: I7263ff1498dfdf51a919aa678adde413c1935bc6
2021-06-03 06:58:44 +00:00
Dan Smith
a854fcfab8 Make taskflow_executor log get_flow() exceptions
Currently an exception raised in a get_flow() task builder will be
silently swallowed with no logging. This patch makes us log it and
then re-raise it so that it will be visible.

Change-Id: I4315bfd68e5a184305f384705e35c4049f75b906
2021-06-02 10:50:03 -07:00
songwenping
3ec92eafc4 Remove references to sys.version_info
We support Python 3.6 as a minimum now, making these checks no-ops.

Change-Id: Ice0409fc063a77f507f25be3e583f79808783e84
2021-06-01 07:08:09 +00:00
Abhishek Kekane
c5c7a2a634 [Doc] Update Freenode to OFTC as our IRC server
Change-Id: I13dd0ada65e882591e3584c04b4bf0e943d1faf1
2021-06-01 04:53:19 +00:00
Zuul
07893401a8 Merge "Cinder Store: Update legacy images tests" 2021-05-31 06:10:45 +00:00
Dan Smith
b8926ab429 Make project_id a formal target alias
This makes ImageTarget integrate the aliasing of project_id to owner,
avoiding the need to do it every time we use ImageTarget. This will
simplify subsequent patches.

Change-Id: Ia9998f638546ac36cd200f0c9002eadd18b71094
2021-05-05 08:44:28 -07:00
Dan Smith
0309814d93 Make wsgi tests use noauth deployment flavor
This gives us a consistent context with roles and project affiliation
for later policy check patches.

Change-Id: I241a81aabe0ca6f8c2b2abacedd86aed9e68c79c
2021-05-05 08:37:13 -07:00
Dan Smith
c366ead5ef Fix test_cache_middleware tests to use auth
This fixes the cache tests (and the 'caching' deployment flavor) to
send authorization headers so that they can be checked by later policy
patches.

Change-Id: I6099c7da24e06595e08fd292a5083327b9f0cc64
2021-05-05 08:37:12 -07:00
Dan Smith
8ac32b8a9e Fix auth info on scrubber tests
The scrubber tests do not pass suitable HTTP headers to pass any
of the policy checks we are going to start adding. This just imports
the base header template from other functional tests so that the
context that is constructed has a proper tenant member arrangement.

Change-Id: I350499b4b3f326ade2c64a2d50e1562c413e449a
2021-05-05 08:30:11 -07:00
Timothy Symanczyk
fdc83a8ede Allow member creation when using db.simple api
Within simple/db/api.py, DATA['members'] was being statically
initialized as a dict.

Inspection of the code revealed a reset function that initialized the
same as a list. Further code inspection seemed to confirm that it was
always intended to be a list.

Updated static initialization to be a list.

Change-Id: Ia53890bcd1161b66a1b365bb3bcc7fc9fbcf6e40
Closes-Bug: #1620833
2021-05-04 21:32:36 +00:00
whoami-rajat
0e2043d51e Cinder Store: Update legacy images tests
update tests for cinder's new attachment API code in glance
cinder store. This leaves a couple of mocks in place for the old
way of doing things so that we can tolerate the old and proposed
changes to glance_store. We can remove those once we require the
newer glance_store version.

Change-Id: If4db20affbba37806f8419b6b30050a32bcc2435
2021-04-19 05:07:53 -04:00
Zuul
1157c19a9c Merge "Guidelines for core reviewers" 2021-04-09 17:00:18 +00:00
Dan Smith
2a0d2303c3 Fix image/tasks API for in-progress tasks
A slight error in the tasks_get_by_image() DB API method resulted in
our excluding in-progress tasks from the returned list. This is
because those tasks have expires_at=NULL, and we were comparing
the expires_at>=$NOW to find unexpired tasks. This makes us check for
"NULL or not expired" instead.

We did have a test asserting the wrong behavior, but it was done to
increase coverage and thus was asserting the behavior of the code and
not the *desired* behavior. This fixes that as well.

Closes-Bug: #1922928
Change-Id: I1b6971888673b64ef60bed8fbcc97bbcbcf5c2ac
2021-04-07 08:10:03 -07:00
Abhishek Kekane
120cb146ab Guidelines for core reviewers
Change-Id: Ia8ff60f8184c533b9fc2a709d33cbdfc02793c41
2021-04-05 05:33:01 +00:00
Zuul
8fc9b65393 Merge "Imported Translations from Zanata" 2021-03-24 07:09:02 +00:00
OpenStack Proposal Bot
3ebeea007e Imported Translations from Zanata
For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html

Change-Id: I42a9c08f49fe323be1fddb62bac394ad0e673c80
2021-03-24 06:29:41 +00:00