129 Commits

Author SHA1 Message Date
Victor Stinner
25319368cc Port replicator to Python 3
* replication_dump(): explicit the JSON encoding (UTF-8) to not rely on
  the locale encoding, otherwise the function will fail on non-ASCII
  character if the locale encoding is ASCII (ex: POSIX locale aka "C")
* Use sorted() to get the sorted list of keys. dict.keys() returns a
  generator on Python 3 which has no sort() method.
* Fix FakeImageService: HTTP body is bytes
* utils.py: get the urllib module from six.moves
* tox.ini: add test_glance_replicator to Python 3.4

Change-Id: Iaded283c70f49abb2408565563c173f58fff2fc7
2015-08-16 11:50:08 +02:00
Victor Stinner
eb769fe477 Port glance.db to Python 3
* add(), save(): don't compare None to an integer, it raises a TypeError
  on Python 3.
* _pop_task_info_values(): copy values items because the values
  dictionary is modified in the loop body.
* replace filter() with a list-comprehension: filter() returns a
  generator on Python 3 which has no length.
* tox.ini: add test_db and test_quota unit tests to Python 3.4

Change-Id: I1494e81b154fc64df814b5f96c15fe032bb5057b
2015-08-16 11:49:50 +02:00
Victor Stinner
eeedd2030d Port image cache to Python 3
* Use bytes for image content
* On Python 3, set_xattr() encodes the text value to UTF-8
* Open files in binary mode, not in text mode
* get_caching_iter(): pass a list as the image iterator, not a string.
  On Python 3, list(b'abc') returns [97, 98, 99], whereas Python 2 returns
  ['a', 'b', 'c'].
* tox.ini: add glance.tests.unit.test_image_cache to Python 3.4

Change-Id: I638525d19c42990852cf45dd416318d9a847c303
2015-08-16 11:49:45 +02:00
Victor Stinner
e14f2da796 Fix Python 3 issues in glance.tests.unit.common
* Open file in text mode to write a paste configuration file, don't open
  in binary mode
* Replace dict.keys()[0] with list(dict.keys())[0], dict.keys() now
  returns an iterator on Python 3
* Replace xrange() with range()
* tox.ini: add the following glance.tests.unit.common tests to Python
  3.4:

  - test_config
  - test_exception
  - test_semver

Change-Id: Ibcfd106cad985b5e08f1e821b3b60a4d0d0911be
2015-08-16 11:39:41 +02:00
Louis Taylor
feb927c8a1 Remove Catalog Index Service
The Catalog Index Service added in the kilo cycle has been split into a new
project named searchlight. This code now lives in a seperate repository:

    https://git.openstack.org/openstack/searchlight

For more information about the split, see the governance change:
I8b44aac03585c651ef8d5e94624f64a0ed2d10b2

DocImpact
UpgradeImpact
APIImpact

Change-Id: I239ac9e32857f6a728f40c169e773ee977cca3ca
2015-07-30 09:34:57 +08:00
Victor Stinner
f05e1e3497 Fix Python 3 issues in glance.tests.unit
* api/versions.py: HTTP body is bytes, encode JSON to UTF-8 on Python 3
* urlsafe_encrypt(): replace str with six.binary_type, encode plaintext
  to UTF-8 if it is Unicode
* urlsafe_decrypt(): replace str with six.binary_type, decode result
  from UTF-8 on Python 3
* MetadefIndex: replace map() with a list comprehension to get a list on
  Python 3, map() returns an iterator on Python 3
* test_artifact_type_definition_framework: skip sort() test on Python 3
  because int and str are not comparable
* test_cache_middleware: HTTP body is bytes, replace '' with b''
* test_misc: add checks on the result type of urlsafe_encrypt() and
  urlsafe_decrypt(). On Python 3, encode plaintext to UTF-8 to compare
  it to ciphertext. Comparing bytes and str raises a TypeError when
  python3 is run with the -bb command line option.
* tox.ini: add to following glance.tests.unit tests to Python 3.4

  - test_artifact_type_definition_framework
  - test_cache_middleware
  - test_db_metadef
  - test_misc
  - test_policy
  - test_search
  - test_versions

Change-Id: Id47c5e9ba761a61d1cb80b65b0b6238f4a331c8c
2015-07-25 11:53:17 +02:00
Zhenguo Niu
94f5096750 Remove H302,H402,H904
These rules were removed from hacking 0.10.x.

Change-Id: I0f6c47c1c87103e85195fac3ed910638d35aaf33
2015-07-17 07:02:48 +00:00
Jeremy Stanley
1114f41edb Remove unneeded OS_TEST_DBAPI_ADMIN_CONNECTION
The desired OS_TEST_DBAPI_ADMIN_CONNECTION string is now reflected
in the oslo.db 1.12.0 release's default behavior, and so does not
need to be set any longer to achieve the same opportunistic backend
discovery for tests.

Change-Id: I73acade8f2b286a3b3fdc25094494e20b4d33afb
2015-07-03 22:59:30 +00:00
Robert Collins
2c4b084a3c Update requirements
- current global requirements
- merge py3 req files into markers

The one eww moment is the -e line for glance store, which only got in
because the infra check-requirements job wasn't checking -py3 files at
all until recently.

Change-Id: I748e9e1bb698085a19c5d8d6060f4d377695dc89
2015-07-02 11:40:46 +12:00
Jenkins
b534954142 Merge "Switch from MySQL-python to PyMySQL" 2015-06-29 12:55:24 +00:00
Victor Stinner
e766b9e47f Fix tox -e py34
* Add requirements-py3.txt: get the master branch of glance_store from
  github until a new release of glance_store including all my Python 3
  fixed will be released.
* Add test-requirements-py3.txt: remove qpid-python and MySQL-python
  (not compatible with Python 3), add PyMySQL
* tox.ini: add py34 test environment running tests with testtools.run.
  Use a whitelist of tests which are known to pass on Python 3.4. testr
  cannot be used because it loads all tests, and not all tests are
  compatible with Python 3.4 yet.

Change-Id: If19af8351e9497db26c9cd62393d22d2043a3903
2015-06-19 14:33:19 +02:00
Jeremy Stanley
98348df246 Switch from MySQL-python to PyMySQL
As discussed in the Liberty Design Summit "Moving apps to Python 3"
cross-project workshop, the way forward in the near future is to
switch to the pure-python PyMySQL library as a default.

https://etherpad.openstack.org/p/liberty-cross-project-python3

Also set the OS_TEST_DBAPI_ADMIN_CONNECTION override variable so
that oslo.db opportunistic detection will know to use PyMySQL until
I12b32dc097a121bd43991bc38dd4d289b65e86c1 makes it the default
behavior.

Change-Id: I66b7d6053643e82d9c11af568190d82ffe7f385e
Co-Authored-By: Victor Sergeyev <vsergeyev@mirantis.com>
2015-06-18 13:53:00 +00:00
Yuiko Takada
9f44574560 Pass environment variables of proxy to tox
When a development environment is under a proxy, tox is failed even if
environment variables of the proxy are set.

This patch fixes this problem.

See this Nova change: If0bc42891a0714cc9b37b9c211388db20555ea7e

Change-Id: I94736408285ad741ff16eab1f9ae9bd30d3f0081
2015-06-15 12:16:09 +00:00
rsritesh
33e418b8f2 Bug : tox -egenconfig failure (no glance-search.conf)
Oslo generator is not able to generate glance-serach.conf.
Removed generation part for glance search from tox file.
Oslo generator is not supporting glance serach namespaces.

Closes-Bug: #1455141

Change-Id: I89236437c1e626df39e1407b430b01f77716e98c
2015-05-25 05:41:40 +02:00
Lakshmi N Sampath
9911f962a4 Catalog Index Service
Implements: blueprint catalog-index-service

* Glance Index and Search API Implementation
* Tool for indexing metadefinition resources and images from Glance
database into Elasticsearch index

Change-Id: I6c27d032dea094c7bf5a30b02100170e265588d9
Co-Authored-By: Lakshmi N Sampath <lakshmi.sampath@hp.com>
Co-Authored-By: Kamil Rykowski <kamil.rykowski@intel.com>
Co-Authored-By: Travis Tripp <travis.tripp@hp.com>
Co-Authored-By: Wayne Okuma <wayne.okuma@hp.com>
Co-Authored-By: Steve McLellan <steve.mclellan@hp.com>
2015-03-26 10:12:27 -07:00
Jenkins
fc90e10f7d Merge "Work toward Python 3.4 support and testing" 2015-02-06 21:55:19 +00:00
Sean Dague
b588479726 remove need for netaddr
The only reason netaddr was pulled in was for ip validation routines
which exist in oslo.utils. Use those instead.

Change-Id: Ic3beee7e8bfb2b2d16ecac9be6322fe2704c70bd
2015-01-15 12:15:51 +00:00
Sean Dague
452dfc65cc remove extraneous --concurrency line in tox
the default testr mode is 1 worker per cpu, which is also what
--concurrency=0 produces. There is no reason to put this in explicitly

Change-Id: I49613ba409f55a21a5392a4276e3f8f6dbc169fe
2015-01-14 22:14:10 +00:00
Zhi Yan Liu
882049a613 Using oslo.concurrency lib
Replace processutils and lockutils modules of oslo-incubator with
oslo.concurrency lib.

Change-Id: Ic1af8753a70f1aada22efe8132e48cbc16e14f3f
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
2014-12-17 13:20:54 +08:00
Doug Hellmann
50a5ffed12 Use testr directly from tox
Invoking testr directly lets us pass arguments to it through tox so we
can, for example, only run a subset of the tests.

Change-Id: I21351880db0848b860e7c61d4a8c711be2a49de0
2014-12-10 14:41:39 -05:00
Jeremy Stanley
c0393cc916 Work toward Python 3.4 support and testing
Change-Id: I0f65d39e93f811892fa79b278eec1335fd5c59eb
2014-11-26 15:54:05 +00:00
Julien Danjou
067b0db570 Remove Python 2.6 classifier
Glance does not support Python 2.6 anymore starting with Kilo and might
not work correctly with it, so remove the classifier.

Change-Id: Ibd7ad401198749f975fb19007549f7623caae546
2014-11-25 17:00:14 +01:00
James Carey
3f2a56ae65 Move from using _ builtin to using glance.i18n _
The builtin _ should not longer be used.  Instead _
should be explicitly assigned from glance.i18n.

This patch does the following:
  (1) adds these explict assignments to any
      glance file that was missing them,
  (2) removes the defintion of _ as a builtin from
      tox.ini so that a pep8 failure will occur
      if _ is not defined,
  (3) removes calls to install().

This removes the last direct use of gettextutils by glance.
The indirect uses, via openstack/common, will be removed
in the next sync with oslo.

Change-Id: Ie54e8a67e747e022cc01022dfad35d89686bdfc4
2014-11-21 18:51:19 +00:00
Julien Danjou
fadbef8511 hacking: upgrade to 0.9.x serie
Change-Id: I252758fd633662de9659a402c5e3d7e3ce1fae0f
2014-10-06 14:41:03 +02:00
Oleksii Chuprykov
868fd6e383 Run tests with default concurrency 0
Setting concurrency to 0 would enable Glance tests to be run using the
the total number of cores on the system rather than by a single one.

This would, by default, speed up the run time of tests for developers
who are not aware about it. Also, any jobs which run these tests for
Continuous Integration would be benefitted without having to deal with
the machine level logic.

Closes-Bug: #1368274

Change-Id: I0842b0c0be1272d479fad9ef7609cd8a29231cb5
2014-09-30 11:37:02 +00:00
Thomas Bechtold
dd9520265b Add specific docs build option to tox
Add doc venv to align with other OpenStack projects (such as keystone,
heat and nova) who are using 'docs' for their doc builds.

Change-Id: I8bccbf9c6b1eae5de9f74396fd5159662ad04149
2014-09-24 00:18:44 +00:00
Zhi Yan Liu
819f28a0b8 Enabling separated sample config file generation
As a common approach most projects used now and Oslo preferred, this
change enabled sample configuration file generation mechanism for
each Glance services.

This change, as an enhancement, allows generating separated sample
configuration files for each Glance major services, e.g.:

    etc/glance-api.conf.sample
    etc/glance-cache.conf.sample
    etc/glance-manage.conf.sample
    etc/glance-registry.conf.sample
    etc/glance-scrubber.conf.sample

It is different than I94d486d6686815c45705a7a9b00fb26062e1eb63
which only supports generating an unified sample configuration
file to including all Glance available options.

This mechanism  not only can help auditing by packager, milestone
maintainer or developer as a function (testenv) of tox, but also
those separated configuration files could make deployment be easy.
And it helps keeping sample configuration files be update with
code change.

The change added "genconfigs" function as a tox testenv (-egenconfigs),
and it dependes on oslo-config-generator function of oslo.config.

The change doesn't introduce those sample files Glance repo, so
next step is to investigate if we can generate them in gate
automatically when a change was merged.

Related-Change-Id: I15686708fc9460948a58cfea3d18dae40ba1fda9
Related-Change-Id: Iae31856d5886ee78786972d80c7c103c3460a2b3
Related-Change-Id: I76043b08e2872867e5af2a5ac902e4d092fda5c8
Closes-Bug: #1300546
Closes-Bug: #1361963

Change-Id: Ibe03a3fe80b96ca32acb1a6bea7e38e6075951bb
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
2014-09-17 13:19:00 +08:00
Julia Varlamova
0c151d7d7e Enable F821 check: undefined name 'name'
Enable F821 and fix usage of undefined variables.

In glance/common/client.py in BaseClient class image_iterator()
function returns instances of unexisting classes:
- SendFileIterator
- ImageBodyIterator
Since we have not these classes functionality now, we will use
utils.chunkreadable() function instead.

Delete image_iterator() function.

Change-Id: Ie1a8c66e5df2836e6e9aadfb1013d69fc6f710e6
2014-07-23 16:28:24 +04:00
Christian Berendt
86dd9ff66c debug level logs should not be translated
According to the OpenStack translation policy available at
https://wiki.openstack.org/wiki/LoggingStandards debug messages
should not be translated. Like mentioned in several changes in
Nova by garyk this is to help prioritize log translation.

This patch adds a new hacking check - N319 - that ensures all
debug log messages don't have translations.

Change-Id: I9dd958b904671a7eb95883026e14684469dc52d5
Closes-Bug: #1317847
2014-05-23 15:57:06 +02:00
Jenkins
1ac9192e96 Merge "Enable H304 check" 2014-04-20 14:48:32 +00:00
Andreas Jaeger
9777660809 Fix Jenkins translation jobs
The job glance-propose-translation-update
does not update from
transifex since our po files contain duplicate entries where
obsolete entries duplicate normal entries.

Remove all obsolete entries to fix the job.

Add test to pep8 that checks that no new breakages get
introduced.

Change-Id: I1bae1c1bf27bf664b24cf75fc96efb0fcfddc392
Closes-Bug: #1299349
2014-04-03 20:13:12 +02:00
Julia Varlamova
5843e071d4 Enable H304 check
Enable H304: no relative imports

Fix import in tools/install_venv.py

Change-Id: I099ed65db9b42223eaa4b66a3a5c6113d1cc56fe
2014-03-20 10:34:26 +00:00
Julia Varlamova
67c79540c7 Enable hacking H301 and disable H304, H302
Enable hacking H301: one import per line. H304 and H302 check enables too,
so we temporarily disable it since it does not pass.

Fix imports in files below.

Add # noqa to migration scripts.

Change-Id: I6a8d4dcd9c3195d2848f218aafe304b1240ab60c
2014-02-18 16:05:18 +04:00
Julia Varlamova
3fa66f63c4 Enable F841 check
Enable F841 check: local variable 'name' assigned but never used.

Make appropriate changes to files listed below.

Change-Id: I02837d4abf421dc9d85f3b01587120fd68acfa12
2014-02-13 16:54:12 +04:00
Jeremy Stanley
00467964c7 Remove tox locale overrides
* tox.ini: The LANG, LANGUAGE and LC_ALL environment overrides were
introduced originally during the testr migration in an attempt to be
conservative about the possibility that locale settings in the
calling environment could cause consistency problems for test runs.
In actuality, this should be unnecessary and any place where it does
cause issues ought to be considered an actual bug. Also, having
these in the configuration actively causes older pip to have
problems with non-ASCII content in some package metadata files under
Python 3, so drop it now.

Change-Id: Ib10f5ef2fd747e38012384f1c6cbe147da8c2573
Closes-Bug: #1277495
2014-02-10 02:59:56 +00:00
Zhi Yan Liu
d7f1221684 Switch to testrepository for running tests
OpenStack as a whole is moving towards using testrepository and
testtools for running tests. To that end, bring Glance into line by
switching it to use testrepository to run tests.

This copies run_tests.sh and tools/colorizer.py from Nova.

This change also has some minor changes to make run_test.sh work well.

Partial fixes bug: 1179009
Fixes bug: 1271806

Change-Id: Ic265bc0d2f1528358f6e8ee5b4139f991923fc72
Signed-off-by: Steve Kowalik <steven@wedontsleep.org>
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
2014-01-29 08:30:55 +08:00
Julia Varlamova
1d0327f13c Enable H302 check
Enable H302 (import only modules) which passes without any changes to files.

Change-Id: Ibcc609ffad404e8d73a4828b7b18892444e7454d
2014-02-07 16:29:36 +04:00
Julia Varlamova
fa3575d8fe Enable H202 check
Enable H202 check: assertRaises Exception too broad.

Make changes to glance/tests/unit/test_notifier.py

Change-Id: I258367172e77312ddda06132d2e8371e8fff81e7
2014-02-06 15:30:47 +04:00
Julia Varlamova
775072583b Enable hacking H703 check
Enable hacking H703 (multiple positional placeholders)
and fix string formatting in files listed below:

- glance/api/authorization.py
- glance/api/v1/images.py
- glance/api/v1/upload_utils.py
- glance/api/v2/image_data.py
- glance/cmd/replicator.py
- glance/common/property_utils.py
- glance/common/utils.py
- glance/common/wsgi.py
- glance/image_cache/__init__.py
- glance/notifier.py
- glance/store/__init__.py
- glance/store/filesystem.py
- glance/store/gridfs.py
- glance/store/swift.py

Change-Id: Idb871eadcad1bd0cc2d2710be7469cf66360c3a7
2014-01-27 14:32:04 +04:00
Dirk Mueller
3eee59d39f Enable gating on H501
H501 - "avoid locals() on string formatting" passes
currently, so no reason to blacklist it.

Change-Id: I9fb1823a598679c0e95c88a8ce4f93629508a80c
2014-01-23 00:15:43 +01:00
Dirk Mueller
17f4c09d0a Switch to Hacking 0.8.x
Blacklist the few hacking warnings that still trigger, those
can be fixed in a followup commit. Start Gating on the new
tests of Hacking 0.8.x (which are all passing here).

Change-Id: I3be1a9a517ec399dd92252a7c705d1a7a6d5e4c2
2014-01-08 00:48:26 +01:00
Dirk Mueller
f0dad5c5d5 Enable gating on F811 (duplicate function definition)
Change-Id: Ifbf233ccbb7cc012d5dae50f9b303cfca592788d
2014-01-02 12:57:15 +01:00
Dirk Mueller
d6114f914c Fix and enable gating on H306
Fix fallout of Hacking check: Imports should be in alphabetical order

Change-Id: I179711c8448208592490ffa47d51419779940014
2014-01-01 16:44:19 +01:00
Yanis Guenane
ec08ca1a87 Make code base E125 and E126 compliant
This commit makes the glance code base E125 and E126 compliant :
  * E125 continuation line does not distinguish itself from next logical line
  * E126 continuation line over-indented for hanging indent

Change-Id: I7120149bedb665fb66320498fe98948602a6cd52
Closes-bug: #1263437
2013-12-30 13:03:24 +00:00
Yanis Guenane
366a68b87d Enable H233/H301/H302 tests that are ignored at the moment
* H233 use of print operator
  * H301 one import per line
  * H302 import only modules

Change-Id: I752c3c117e2575b30cc96d68c18df365d595588e
Closes-bug: #1263535
2013-12-23 23:57:10 +00:00
Yanis Guenane
317ad31637 Make Glance code base H102 compliant
Add the Apache headers where it was missing so the Glance
code base can be H102 compliant

Change-Id: If19841d1752f7ec364098333429ce090374e79cd
Closes-bug: #1263537
2013-12-23 13:12:12 +00:00
Yanis Guenane
70cb81e929 Make Glance code base H201 compliant
Add Exception to all except without any specific exception
class so the code can be H201 compliant

Change-Id: Ibd09e69f70c76837c72023ae87cbbe75c5eb1276
2013-12-23 00:38:52 +00:00
Dirk Mueller
2fb875f79f Fix and enable gating on H702
Formatting operations should be outside of localization,
otherwise translation lookup will never find a translated
string as a match. This also fixes H701 alongway.

Change-Id: I6708e0f5b11841b18ba9042ac4958643f91396dd
2013-12-11 22:12:11 +01:00
Victor Sergeyev
88bb1af1d5 Enable F40X checking
Clean up imports due to F401 and F403 checks.
F401  'module' imported but unused
F403  unable to detect undefined names with wildcard import

Change-Id: I487edb157de1a6babc7ad8a3fb65f195e476c490
2013-11-27 18:15:48 +02:00
Luis A. Garcia
1236af6039 Enable H501: do not use locals() for formatting
Change-Id: I8b63c76b50cb3bb74b136678a945df79eb19856e
2013-11-12 00:50:49 +00:00