Revise database rolling upgrade documentation

- mark zero-downtime-db-upgrade as EXPERIMENTAL for queens
- clarify the relation between the E-M-C strategy and
  zero-downtime db upgrades
- add note that for MySQL, using the glance-manage expand or
  glance-manage contract command requires that the glance
  is granted SUPER privileges
- add note to contributor docs about checking the trigger
  flag in expand and contract scripts

Co-authored-by: Abhishek Kekane <akekane@redhat.com>
Co-authored-by: Brian Rosmaita <rosmaita.fossdev@gmail.com>

Change-Id: I5af4a1428b89ecb05a1be9c420c5f0afc05b9a95
Closes-Bug: #1750555

doc/source/admin/rollingupgrades.rst

@@ -19,6 +19,23 @@ Rolling Upgrades
 .. note:: The Rolling Upgrades feature is EXPERIMENTAL and its use in
           production systems is currently **not supported**.
 
+          This statement remains true for the Queens release of Glance.  What
+          is the holdup, you ask?  Before asserting that the feature is fully
+          supported, the Glance team needs to have automated tests that perform
+          rolling upgrades in the OpenStack Continuous Integration gates.  The
+          Glance project team has not had sufficient testing and development
+          resources in recent cycles to prioritize this work.
+
+          The Glance project team is committed to the stability of Glance.  As
+          part of OpenStack, we are committed to `The Four Opens`_.  If the
+          ability to perform rolling upgrades in production systems is
+          important to you, feel free to participate in the Glance community to
+          help coordinate and drive such an effort.  (We gently remind you that
+          "participation" includes providing testing and development
+          resources.)
+
+          .. _`The Four Opens`: https://governance.openstack.org/tc/reference/opens.html
+
 Scope of this document
 ----------------------
 
@@ -83,6 +100,12 @@ Following is the process to upgrade Glance with zero downtime:
 
     glance-manage db expand
 
+    .. warning::
+
+     For MySQL, using the ``glance-manage db_expand`` command requires that
+     you either grant your glance user ``SUPER`` privileges, or run
+     ``set global log_bin_trust_function_creators=1;`` in mysql beforehand.
+
 5. Then, also on the NEW NODE, perform the data migrations using the command::
 
     glance-manage db migrate

doc/source/admin/zero-downtime-db-upgrade.rst

@@ -17,11 +17,11 @@ Zero-Downtime Database Upgrades
 ===============================
 
 .. warning::
-   This feature is EXPERIMENTAL in the Ocata and Pike releases.
+   This feature is EXPERIMENTAL in the Ocata, Pike and Queens releases.
    We encourage operators to try it out, but its use in production
    environments is currently NOT SUPPORTED.
 
-A zero-downtime database upgrade enables true rolling upgrades of the Glance
+A *zero-downtime database upgrade* enables true rolling upgrades of the Glance
 nodes in your cloud's control plane.  At the appropriate point in the upgrade,
 you can have a mixed deployment of release *n* (for example, Ocata) and release
 *n-1* (for example, Newton) Glance nodes, take the *n-1* release nodes out of
@@ -31,6 +31,13 @@ leaving all Glance nodes in your cloud at release *n*.
 That's a rough sketch of how a rolling upgrade would work.  For full details,
 see :ref:`rolling-upgrades`.
 
+.. note::
+   When we speak of a "database upgrade", we are simply talking about changing
+   the database schema and its data from the version used in OpenStack release
+   *n* (say, Pike) to the version used in OpenStack release *n+1* (say,
+   Queens).  We are **not** talking about upgrading the database management
+   software.
+
 .. note::
    Downgrading a database is not supported.  See :ref:`downgrades` for more
    information.
@@ -38,18 +45,45 @@ see :ref:`rolling-upgrades`.
 The Expand-Migrate-Contract Cycle
 ---------------------------------
 
-For Glance, a zero-downtime database upgrade has three phases:
+It's possible to characterize three phases of a database upgrade:
 
-1. **Expand**: in this phase, new columns, tables, indexes, or triggers are
+1. **Expand**: in this phase, new columns, tables, indexes, are added to the
-   added to the database.
+   database.
 
 2. **Migrate**: in this phase, data is migrated to the new columns or tables.
 
-3. **Contract**: in this phase, the "old" tables or columns (and any database
+3. **Contract**: in this phase, the "old" tables or columns (which are no
-   triggers used during the migration), which are no longer in use, are removed
+   longer in use) are removed from the database.
-   from the database.
 
-The above phases are abbreviated as an **E-M-C** database upgrade.
+The "legacy" Glance database migrations performed these phases as part of a
+single monolithic upgrade script.  Currently, the Glance project creates a
+separate script for each the three parts of the cycle.  We call such an
+upgrade an **E-M-C** database migration.
+
+Zero-Downtime Database Upgrade
+------------------------------
+
+The E-M-C strategy can be performed offline when Glance is not using the
+database.  With some adjustments, however, the E-M-C strategy can be applied
+online when the database is in use, making true rolling upgrades possible.
+
+.. note::
+   Don't forget that zero-downtime database upgrades are currently considered
+   experimental and their use in production environments is NOT SUPPORTED.
+
+A zero-downtime database upgrade takes place as part of a :ref:`rolling upgrade
+strategy <rolling-upgrades>` for upgrading your entire Glance installation.  In
+such a situation, you want to upgrade to release *n* of Glance (say, Queens)
+while your release *n-1* API nodes are still running Pike.  To make this
+possible, in the **Expand** phase, database triggers can be added to the
+database to keep the data in "old" and "new" columns synchronized.  Likewise,
+after all data has been migrated and all Glance nodes have been updated to
+release *n* code, these triggers are deleted in the **Contract** phase.
+
+.. note::
+   Unlike the E-M-C scripts, database triggers are particular to each database
+   technology.  That's why the Glance project currently provides experimental
+   support only for MySQL.
 
 New Database Version Identifiers
 --------------------------------
@@ -59,9 +93,8 @@ database becomes more complicated since it must reflect knowledge of what point
 in the E-M-C cycle the upgrade has reached.  To make this evident, the
 identifier explicitly contains 'expand' or 'contract' as part of its name.
 
-Thus the ``ocata01`` migration (that is, the migration that's currently used in
+Thus the Ocata cycle migration has two identifiers associated with it:
-the fully supported upgrade path) has two identifiers associated with it for
+``ocata_expand01`` and ``ocata_contract01``.
-zero-downtime upgrades: ``ocata_expand01`` and ``ocata_contract01``.
 
 During the upgrade process, the database is initially marked with
 ``ocata_expand01``.  Eventually, after completing the full upgrade process, the
@@ -100,12 +133,24 @@ version record would go through the following progression:
 Database Upgrade
 ----------------
 
-In order to enable the E-M-C database upgrade cycle, and to enable Glance
+For offline database upgrades, the ``glance-manage`` tool still has the
-rolling upgrades, the ``glance-manage`` tool has been augmented to include the
+``glance-manage db sync`` command.  This command will execute the expand,
-following operations.
+migrate, and contract scripts for you, just as if they were contained in
+a single script.
+
+In order to enable zero-downtime database upgrades, the ``glance-manage`` tool
+has been augmented to include the following operations so that you can
+explicitly manage the upgrade.
+
+.. warning::
+
+     For MySQL, using the ``glance-manage db expand`` or
+     ``glance-manage db contract`` command requires that
+     you either grant your glance user ``SUPER`` privileges, or run
+     ``set global log_bin_trust_function_creators=1;`` in mysql beforehand.
 
 Expanding the Database
-----------------------
+~~~~~~~~~~~~~~~~~~~~~~
 ::
 
     glance-manage db expand
@@ -116,7 +161,7 @@ any new services are started.
 
 
 Migrating the Data
-------------------
+~~~~~~~~~~~~~~~~~~
 ::
 
     glance-manage db migrate
@@ -127,7 +172,7 @@ are started.
 
 
 Contracting the Database
-------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~
 ::
 
     glance-manage db contract

doc/source/contributor/database_migrations.rst

@@ -282,10 +282,21 @@ Data Migrations
 NOTES
 -----
 
-* Starting in Ocata, Glance needs every database migration to include both
+* In Ocata and Pike, Glance required every database migration to include
-  monolithic and Expand-Migrate-Contract (E-M-C) style migrations. At some
+  both monolithic and Expand-Migrate-Contract (E-M-C) style migrations.  In
-  point in Pike, E-M-C migrations will be made default. At that point, it
+  Queens, E-M-C migrations became the default and a monolithic migration
-  would be no longer required to include monolithic migration script.
+  script is no longer required.
+
+  In Queens, the glance-manage tool was refactored so that the ``glance-manage
+  db sync`` command runs the expand, migrate, and contract scripts "under
+  the hood".  From the viewpoint of the operator, there is no difference
+  between having a single monolithic script and having three scripts.
+
+  Since we are using the same scripts for offline and online (zero-downtime)
+  database upgrades, as a developer you have to pay attention in your scripts
+  to determine whether you need to add/remove triggers in the expand/contract
+  scripts.  See the changes to the ocata scripts in
+  https://review.openstack.org/#/c/544792/ for an example of how to do this.
 
 * Alembic is a database migration engine written for SQLAlchemy. So, any
   migration script written for SQLAlchemy Migrate should work with Alembic as