Drop namespace packages
Stop using namespace packages for our libraries. Change-Id: I85fa9b834600d3b12450fe29e39122b75d074c65
This commit is contained in:
Doug Hellmann
parent
50bb32b95c
commit
d5f4a91e0e
248
specs/kilo/drop-namespace-packages.rst
Normal file
248
specs/kilo/drop-namespace-packages.rst
Normal file
@ -0,0 +1,248 @@
|
|||||||
|
====================================
|
||||||
|
Drop our use of namespace packages
|
||||||
|
====================================
|
||||||
|
|
||||||
|
https://blueprints.launchpad.net/oslo-incubator/+spec/drop-namespace-packages
|
||||||
|
|
||||||
|
Installing our libraries into a namespace package has caused several
|
||||||
|
types of errors. Rather than continuing to fight with them, I propose
|
||||||
|
that we phase them out.
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
The namespace package support in setuptools is fragile, and some of
|
||||||
|
the ways we use code regularly within OpenStack expose the issues in
|
||||||
|
ways that are hard to debug.
|
||||||
|
|
||||||
|
The main issue we have seen is with installing two separate libraries
|
||||||
|
into the same python namespace package using different installation
|
||||||
|
"modes". If one library installed with ``pip install -e`` to enable
|
||||||
|
"editable" mode, and another is installed without the ``-e`` option as
|
||||||
|
a regular library, then the import path for the package is broken and
|
||||||
|
some of the installed components are not importable. This happened a
|
||||||
|
lot when devstack installed Oslo libraries editable, and we have
|
||||||
|
changed devstack to stop doing that by default. It also happens when a
|
||||||
|
developer installs something using a system package and then installs
|
||||||
|
another library from source. We have no real way to control that
|
||||||
|
behavior, so it still crops up from time to time.
|
||||||
|
|
||||||
|
We also see trouble with virtualenvs configured to allow access to the
|
||||||
|
global site-packages directory, as nova needs for libvirt access. If
|
||||||
|
anything from the ``oslo`` namespace package is installed globally, it
|
||||||
|
will shadow the versions visible in the virtualenv. That would
|
||||||
|
normally lead to a version error if some old version is installed
|
||||||
|
globally, but in this case it can also lead to import errors because
|
||||||
|
the namespace package doesn't not properly span the two site-packages
|
||||||
|
directories.
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
I propose that we move all current Oslo libraries out of the ``oslo``
|
||||||
|
namespace and create simple packages that can be imported
|
||||||
|
independently. For example, we would change::
|
||||||
|
|
||||||
|
from oslo.foo import bar
|
||||||
|
|
||||||
|
to::
|
||||||
|
|
||||||
|
from oslo_foo import bar
|
||||||
|
|
||||||
|
To avoid issues with distros, we should not rename the ``oslo.foo``
|
||||||
|
library to ``oslo_foo``. This may be a little confusing for
|
||||||
|
developers, so we might look at the decision again in the future, but
|
||||||
|
as a first step this is less obtrusive because it means we don't need
|
||||||
|
to change our requirements lists, repository names, and distro package
|
||||||
|
names.
|
||||||
|
|
||||||
|
To support backwards compatibility, we will provide shadow-packages
|
||||||
|
for the ``oslo`` namespace so that the old import form still works for
|
||||||
|
the Kilo releases of the libraries. That will give us time to update
|
||||||
|
all of the applications and let them cycle out of the long-term
|
||||||
|
support window before dropping the namespace packages entirely during
|
||||||
|
the M release.
|
||||||
|
|
||||||
|
Moving to regular packages will also let us move our unit tests inside
|
||||||
|
the library, so they are delivered and installed under ``oslo_foo``,
|
||||||
|
so we should do that at the same time, retaining just enough of the
|
||||||
|
test suite to ensure that the public API exposed through the namespace
|
||||||
|
package still works.
|
||||||
|
|
||||||
|
The distribution names of existing libraries won't change, and the
|
||||||
|
same pattern will be used to create new libraries to avoid
|
||||||
|
confusion. That means one would ``pip install oslo.foo`` and then
|
||||||
|
``from oslo_foo import bar``.
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
1. lifeless has a patch to oslo.db that sets up the namespace package
|
||||||
|
using pkgutils instead of setuptools:
|
||||||
|
|
||||||
|
https://review.openstack.org/#/c/123604/3/oslo/__init__.py
|
||||||
|
|
||||||
|
This retains the ability to use namespace packages, with the extra
|
||||||
|
cost of shipping a real ``oslo`` library
|
||||||
|
(https://github.com/rbtcollins/oslo) and maintaining the override
|
||||||
|
code ourselves. We don't know what other issues we might find with
|
||||||
|
namespace packages, though, so I'm more comfortable with just
|
||||||
|
dropping them entirely.
|
||||||
|
|
||||||
|
2. Python 3 has better native namespace package support, so we could
|
||||||
|
review the decision when we move to Python 3.
|
||||||
|
|
||||||
|
3. Do nothing, and continue to help developers debug problems on their
|
||||||
|
systems when they come up. If we choose this option, we might want
|
||||||
|
to write a script "why_is_oslo_broken.sh" to put in the incubator.
|
||||||
|
|
||||||
|
The "oslo\_" name prefix was selected to distinguish the libs from the
|
||||||
|
non-production libraries like oslotest and oslosphinx. We could
|
||||||
|
theoretially rename those as well so that all of our branded libraries
|
||||||
|
have the same naming convention. I don't know if that's worth doing or
|
||||||
|
not.
|
||||||
|
|
||||||
|
Impact on Existing APIs
|
||||||
|
-----------------------
|
||||||
|
|
||||||
|
The imports will change, but we should not need to change other
|
||||||
|
aspects of the libraries' APIs.
|
||||||
|
|
||||||
|
Security impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Performance Impact
|
||||||
|
------------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Configuration Impact
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Developer Impact
|
||||||
|
----------------
|
||||||
|
|
||||||
|
Developers need to be aware of the import changes. We have a few
|
||||||
|
options for communicating that widely:
|
||||||
|
|
||||||
|
1. Add a deprecation warning to the oslo namespace packages so
|
||||||
|
importing them reports a warning.
|
||||||
|
2. Add a hacking rule to make importing "from oslo." illegal.
|
||||||
|
3. Rely on reviewers to catch it.
|
||||||
|
|
||||||
|
Option 1 is easy for us to do ourselves.
|
||||||
|
|
||||||
|
Option 2 has to wait until we have converted all of our own libraries,
|
||||||
|
and updated all the apps, but it would then prevent improper imports
|
||||||
|
being restored.
|
||||||
|
|
||||||
|
Option 3 is brittle, so I don't think we want to rely on it alone.
|
||||||
|
|
||||||
|
To ease the transition, we should be able to prepare a bash/sed script
|
||||||
|
to make the required edits to a project. I don't think we'll have
|
||||||
|
more than a handful of different types of import statements, but we
|
||||||
|
can update the script as we find new patterns
|
||||||
|
|
||||||
|
Testing Impact
|
||||||
|
--------------
|
||||||
|
|
||||||
|
The unit tests will be updated in each library so that all of them run
|
||||||
|
against the new module name. Then some of the public API tests will be
|
||||||
|
duplicated to use the namespace package to ensure that we don't break
|
||||||
|
that.
|
||||||
|
|
||||||
|
The existing unit tests in applications should cover the uses of the
|
||||||
|
libraries in the applications. We will need to update any mocks in
|
||||||
|
those tests.
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
Doug Hellmann
|
||||||
|
|
||||||
|
Other contributors:
|
||||||
|
None
|
||||||
|
|
||||||
|
Milestones
|
||||||
|
----------
|
||||||
|
|
||||||
|
Target Milestone for completion: K-2 (I hope early)
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
1. Rearrange all of our library code, including tests and
|
||||||
|
documentation. See https://review.openstack.org/#/c/127323/ for an
|
||||||
|
example.
|
||||||
|
2. Write helper script for liaisons (maybe a liaison can do this?).
|
||||||
|
|
||||||
|
Incubation
|
||||||
|
==========
|
||||||
|
|
||||||
|
N/A
|
||||||
|
|
||||||
|
Adoption
|
||||||
|
--------
|
||||||
|
|
||||||
|
N/A
|
||||||
|
|
||||||
|
Library
|
||||||
|
-------
|
||||||
|
|
||||||
|
N/A
|
||||||
|
|
||||||
|
Anticipated API Stabilization
|
||||||
|
-----------------------------
|
||||||
|
|
||||||
|
N/A
|
||||||
|
|
||||||
|
Documentation Impact
|
||||||
|
====================
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
* setuptools bug 250, "develop and
|
||||||
|
`--install-single-version-externally-managed` are not compatible
|
||||||
|
with each other for namespace packages" -
|
||||||
|
https://bitbucket.org/pypa/setuptools/issue/250/develop-and-install-single-version
|
||||||
|
|
||||||
|
* lifeless has a patch to oslo.db that sets up the namespace package
|
||||||
|
using pkgutils instead of setuptools:
|
||||||
|
|
||||||
|
https://review.openstack.org/#/c/123604/3/oslo/__init__.py
|
||||||
|
|
||||||
|
it uses a real oslo package: https://github.com/rbtcollins/oslo
|
||||||
|
|
||||||
|
* PEP-420 spec: http://legacy.python.org/dev/peps/pep-0420/
|
||||||
|
|
||||||
|
* My WIP patch for oslo.i18n to move it out of the namespace package:
|
||||||
|
https://review.openstack.org/#/c/127323/
|
||||||
|
|
||||||
|
* Notes from the Kilo summit session:
|
||||||
|
https://etherpad.openstack.org/p/kilo-oslo-namespace-packages
|
||||||
|
|
||||||
|
* Mailing list thread following up after the Kilo summit:
|
||||||
|
http://lists.openstack.org/pipermail/openstack-dev/2014-November/050313.html
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0
|
||||||
|
Unported License.
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
Loading…
Reference in New Issue
Block a user