From e721550e062fedb3783916c538eec80972e52a6b Mon Sep 17 00:00:00 2001
From: Beth Elwell <belwell@redhat.com>
Date: Thu, 23 Feb 2017 09:59:47 -0500
Subject: [PATCH] Amended testing documentation

Added documentation for Jasmine break points and how to check why
tests fail minimum threshold. Also removed irrelevant python test
documentation.

Change-Id: Idf3b4bf74aaf4712a13c3c3e4c6f0880457f78d8
---
 doc/source/index.rst                     |   2 +-
 doc/source/topics/javascript_testing.rst |  15 +-
 doc/source/topics/testing.rst            | 446 ++---------------------
 3 files changed, 38 insertions(+), 425 deletions(-)

diff --git a/doc/source/index.rst b/doc/source/index.rst
index e70dd95d59..ca54b128c5 100644
--- a/doc/source/index.rst
+++ b/doc/source/index.rst
@@ -92,8 +92,8 @@ the following topic guides.
    topics/workflows
    topics/tables
    topics/policy
-   topics/testing
    topics/angularjs
+   topics/testing
    topics/javascript_testing
    topics/styling
    topics/translation
diff --git a/doc/source/topics/javascript_testing.rst b/doc/source/topics/javascript_testing.rst
index 1fd702d34a..71b8a35767 100644
--- a/doc/source/topics/javascript_testing.rst
+++ b/doc/source/topics/javascript_testing.rst
@@ -39,6 +39,16 @@ Tests can be run in two ways:
 The code linting job can be run with ``tox -e npm -- lint``, or
 ``tox -e npm -- lintq`` to show errors, but not warnings.
 
+To decipher where tests are failing it may be useful to use Jasmine in the
+browser to run individual tests to see where the tests are specifically breaking.
+To do this, navigate to your local horizon in the browser and add
+'/jasmine' to the end of the url. e.g: 'http://localhost:8000/jasmine'. Once you
+have the jasmine report you may click on the title of an individual test to
+re-run just that test. From here, you can also use chrome dev tools or similar
+to set breakpoints in the code by accessing the 'Sources' tab and clicking on
+lines of code where you wish to break the code. This will then show you the exact
+places where the code breaks.
+
 Coverage Reports
 ----------------
 
@@ -47,9 +57,8 @@ developing, be sure to check the coverage reports on the master branch and
 compare your development branch; this will help identify missing tests.
 
 To generate coverage reports, run ``tox -e npm``. The coverage reports can be
-found at ``horizon/coverage-karma/`` (framework tests) and
-``openstack_dashboard/coverage-karma/`` (dashboard tests). Load
-``<browser>/index.html`` in a browser to view the reports.
+found at ``cover/horizon/`` (framework tests) and ``cover/openstack_dashboard/``
+(dashboard tests). Load ``<browser>/index.html`` in a browser to view the reports.
 
 Writing Tests
 =============
diff --git a/doc/source/topics/testing.rst b/doc/source/topics/testing.rst
index 45153fb2d9..46e96beade 100644
--- a/doc/source/topics/testing.rst
+++ b/doc/source/topics/testing.rst
@@ -1,6 +1,6 @@
-===================
-Testing Topic Guide
-===================
+================
+Testing Overview
+================
 
 Having good tests in place is absolutely critical for ensuring a stable,
 maintainable codebase. Hopefully that doesn't need any more explanation.
@@ -13,6 +13,11 @@ If you already know everything about testing but are fed up with trying to
 debug why a specific test failed, you can skip the intro and jump
 straight to :ref:`debugging_unit_tests`.
 
+.. toctree::
+  :maxdepth: 1
+
+  Angular specific testing <javascript_testing>
+
 An overview of testing
 ======================
 
@@ -222,6 +227,23 @@ Tips and tricks
     +        },
              'django.db.backends': {
 
+Coverage reports
+----------------
+
+It is possible for tests to fail on your patch due to the npm-run-test not
+passing the minimum threshold. This is not necessarily related directly to the
+functions in the patch that have failed, but more that there are not enough tests
+across horizon that are related to your patch.
+
+The coverage reports may be found in the 'cover' directory. There's a subdirectory
+for horizon and openstack_dashboard, and then under a directory for the browser
+used to run the tests you should find an ``index.html``. This can then be viewed
+to see the coverage details.
+
+In this scenario you may need to submit a secondary patch to address test coverage
+for another function within horizon to ensure tests rise above the coverage
+threshold and your original patch can pass the necessary tests.
+
 Common pitfalls
 ---------------
 
@@ -292,421 +314,3 @@ result of an error in the conditions of the test. Using the
 :meth:`~horizon.test.helpers.TestCase.assertMessageCount` will make it readily
 apparent what the problem is in the majority of cases. If not, then use ``pdb``
 and start interrupting the code flow to see where things are getting off track.
-
-Integration tests in Horizon
-============================
-
-The integration tests currently live in the Horizon repository, see `here`_,
-which also contains instructions on how to run the tests. To make integration
-tests more understandable and maintainable, the Page Object pattern is used
-throughout them.
-
-.. warning:: To enable integration tests support before running them, please
-   copy openstack_dashboard/local/local_settings.d/_20_integration_tests_scaffolds.py.example
-   to openstack_dashboard/local/local_settings.d/_20_integration_tests_scaffolds.py
-   and then run ./manage.py collectstatic --clear && ./manage.py compress.
-
-Horizon repository also provides two shell `scripts`_, which are executed in
-pre_test_hook and post_test_hook respectively. Pre hook is generally used for
-modifying test environment, while post hook is used for running actual
-integration tests with tox and collecting test artifacts. Thanks to the
-incorporating all modifications to tests into Horizon repository, one can alter
-both tests and test environment and see the immediate results in Jenkins job
-output.
-
-.. _here: https://github.com/openstack/horizon/tree/master/openstack_dashboard/test/integration_tests
-.. _scripts: https://github.com/openstack/horizon/tree/master/tools/gate/integration
-
-Page Object pattern
--------------------
-
-Within any web application's user interface (UI) there are areas that the tests
-interact with. A Page Object simply models these as objects within the test
-code. This reduces the amount of duplicated code; if the UI changes, the fix
-needs only be applied in one place.
-
-Page Objects can be thought of as facing in two directions simultaneously.
-Facing towards the developer of a test, they represent the services offered by
-a particular page. Facing away from the developer, they should be the only
-thing that has a deep knowledge of the structure of the HTML of a page (or
-part of a page). It is simplest to think of the methods on a Page Object as
-offering the "services" that a page offers rather than exposing the details
-and mechanics of the page. As an example, think of the inbox of any web-based
-email system. Amongst the services that it offers are typically the ability to
-compose a new email, to choose to read a single email, and to list the subject
-lines of the emails in the inbox. How these are implemented should not matter
-to the test.
-
-Writing reusable and maintainable Page Objects
-----------------------------------------------
-
-Because the main idea is to encourage the developer of a test to try and think
-about the services that they are interacting with rather than the
-implementation, Page Objects should seldom expose the underlying WebDriver
-instance. To facilitate this, methods on the Page Object should return other
-Page Objects. This means that we can effectively model the user's journey
-through the application.
-
-Another important thing to mention is that a Page Object need not represent an
-entire page. It may represent a section that appears many times within a site
-or page, such as site navigation. The essential principle is that there is
-only one place in your test suite with knowledge of the structure of the HTML
-of a particular (part of a) page. With this in mind, a test developer builds
-up regions that become reusable components (`example of a base form`_). These
-properties can then be redefined or overridden (e.g. selectors) in the actual
-pages (subclasses) (`example of a tabbed form`_).
-
-The page objects are read-only and define the read-only and clickable elements
-of a page, which work to shield the tests. For instance, from the test
-perspective, if "Logout" used to be a link but suddenly becomes an option in a
-drop-down menu, there are no changes (in the test itself) because it still simply
-calls the "click_on_logout" action method.
-
-This approach has two main aspects:
-
-* The classes with the actual tests should be as readable as possible
-* The other parts of the testing framework should be as much about data as
-  possible, so that if the CSS etc. changes you only need to change that one
-  property. If the flow changes, only the action method should need to change.
-
-There is little that is Selenium-specific in the Pages, except for the
-properties. There is little coupling between the tests and the pages. Writing
-the tests becomes like writing out a list of steps (by using the previously
-mentioned action methods). One of the key points, particularly important for
-this kind of UI driven testing is to isolate the tests from what is behind
-them.
-
-.. _example of a base form: https://github.com/openstack/horizon/blob/8.0.0/openstack_dashboard/test/integration_tests/regions/forms.py#L250
-.. _example of a tabbed form: https://github.com/openstack/horizon/blob/8.0.0/openstack_dashboard/test/integration_tests/regions/forms.py#L322
-
-List of references
-------------------
-
-* https://wiki.openstack.org/wiki/Horizon/Testing/UI#Page_Object_Pattern_.28Selected_Approach.29
-* https://wiki.mozilla.org/QA/Execution/Web_Testing/Docs/Automation/StyleGuide#Page_Objects
-* https://code.google.com/p/selenium/wiki/PageObjects
-
-Debugging integration tests
-===========================
-
-Even perfectly designed Page Objects are not a guarantee that your integration
-test will not ever fail. This can happen due to different causes:
-
-The first and most anticipated kind of failure is the inability to perform a
-testing scenario by a living person simply because some OpenStack service or
-Horizon itself prevents them from doing so. This is exactly the kind that
-integration tests are designed to catch. Let us call them "good" failures.
-
-All other kinds of failures are unwanted and could be roughly split into the
-two following categories:
-
-#. The failures that occur due to changes in application's DOM. some CSS/ Xpath selectors no longer matching
-   Horizon app's DOM. The usual signature for that kind of failures is having
-   a DOM changing patch for which the test job fails with a message like
-   this `selenium.common.exceptions.NoSuchElementException: Message: Unable to
-   locate element: {"method":"css selector","selector":"div.modal-dialog"}`.
-   If you find yourself in such a situation, you should fix the Page Object
-   selectors according to the DOM changes you made.
-
-#. Unfortunately it is still quite possible to get the above error for a patch
-   which didn't implement any DOM changes. Among the reasons of such behavior
-   observed in past were:
-
-   * Integration tests relying on relative ordering of form fields and table
-     actions that broke with the addition of a new field. This issue should
-     be fixed by now, but may reappear in future for different entities.
-
-   * Integration tests relying on popups disappearing by the time a specific
-     action needs to be taken (or not existing at all). This expectation
-     turned out to be very fragile, since the speed of tests execution by
-     Jenkins workers may change independently of integration test code (hence,
-     popups disappear too late to free the way for the next action). The
-     unexpected (both too long and too short) timeouts aren't limited to just
-     popups, but apply to every situation when the element state transition
-     is not instant (like opening an external link, going to another page in
-     Horizon, waiting for button to become active, waiting for a table row to
-     change its state). Luckily, most transitions of "element becomes visible/
-     emerge to existence from non-existence" kind are already bulletproofed
-     using `implicit_wait` parameter in `integration_tests/horizon.conf` file.
-     Selenium just waits for specified amount of seconds for an element to
-     become visible (if it's not already visible) giving up when it exceeds
-     (with the above error). Also it's worth mentioning `explicit_wait` parameter
-     which is considered when the selenium `wait_until` method is involved (and
-     it is used, e.g. in waiting for spinner and messages popups to disappear).
-
-An inconvenient thing about reading test results in the `console.html` file
-attached to every `gate-horizon-dsvm-integration` finished job is that the test
-failure may appear either as failure (assertion failed), or as error (expected
-element didn't show up). In both cases an inquirer should suspect a legitimate
-failure first (i.e., treat errors as failures). Unfortunately, no clear method
-exists for the separation of "good" from "bad" failures. Each case is
-unique and full of mysteries.
-
-The Horizon testing mechanism tries to alleviate this ambiguity by providing
-several facilities to aid in failure investigation:
-
-* First there comes a screenshot made for every failed test (in a separate
-  folder, on a same level as `console.html`) - almost instant snapshot of a
-  screen on the moment of failure (*almost* sometimes matters, especially in
-  a case of popups that hang on a screen for a limited time);
-* Then the patient inquirer may skim through the vast innards of
-  `console.html`, looking at browser log first (all javascript and css errors
-  should come there),
-* Then looking at a full textual snapshot of a page for which test failed
-  (sometimes it gives a more precise picture than a screenshot),
-* And finally looking at test error stacktrace (most useful) and a lengthy
-  output of requests/ responses with a selenium server. The last log sometimes
-  might tell us how long a specific web element was polled before failing (in
-  case of `implicit_wait` there should be a series of requests to the same
-  element).
-
-The best way to solve the cause of test failure is running and debugging the
-troublesome test locally. You could use `pdb` or Python IDE of your choice to
-stop test execution in arbitrary points and examining various Page Objects
-attributes to understand what they missed. Looking at the real page structure
-in browser developer tools also could explain why the test fails. Sometimes it
-may be worth to place breakpoints in JavaScript code (provided that static is
-served uncompressed) to examine the objects of interest. If it takes long, you
-may also want to increase the webdriver's timeout so it will not close browser
-windows forcefully. Finally, sometimes it may make sense to examine the
-contents of `logs` directory, especially apache logs - but that is mostly the
-case for the "good" failures.
-
-Writing your first integration test
-===================================
-
-So, you are going to write your first integration test and looking for some
-guidelines on how to do it. The first and the most comprehensive source of
-knowledge is the existing codebase of integration tests. Look how other tests
-are written, which Page Objects they use and learn by copying. Accurate imitation
-will eventually lead to a solid understanding. Yet there are few things that may
-save you some time when you know them in advance.
-
-File and directory layout and go_to_*page() methods
----------------------------------------------------
-Below is the filesystem structure that test helpers rely on.::
-
-  horizon/
-  └─ openstack_dashboard/
-     └─ test/
-        └─ integration_tests/
-           ├─ pages/
-           │  ├─ admin/
-           │  │  ├─ __init__.py
-           │  │  └─ system/
-           │  │     ├─ __init__.py
-           │  │     └─ flavorspage.py
-           │  ├─ project/
-           │  │  └─ compute/
-           │  │     ├─ __init__.py
-           │  │     ├─ access_and_security/
-           │  │     │  ├─ __init__.py
-           │  │     │  └─ keypairspage.py
-           │  │     └─ imagespage.py
-           │  └─ navigation.py
-           ├─ regions/
-           ├─ tests/
-           ├─ config.py
-           └─ horizon.conf
-
-New tests are put into integration_tests/tests, where they are grouped
-by the kind of entities being tested (test_instances.py, test_networks.py, etc).
-All Page Objects to be used by tests are inside pages/directory, the nested
-directory structure you see within it obeys the value of `Navigation.CORE_PAGE_STRUCTURE`
-you can find at pages/navigation.py module. The contents of the `CORE_PAGE_STRUCTURE`
-variable should in turn mirror the structure of standard dashboard sidebar menu.
-If this condition is not met, the go_to_<pagename>page() methods which are generated
-automatically at runtime will have problems matching the real sidebar items. How are
-these go_to_*page() methods are generated? From the sidebar's point of view, dashboard
-content could be at most four levels deep: Dashboard, Panel Group, Panel and Tab.
-Given the mixture of these entities in existing dashboards, it was decided that:
-
-* When panels need to be addressed with go_to_<pagename>page() methods, two components in
-  the method's name are enough for distinguishing the right path to go along, namely a Panel
-  name and a Panel Group name (or a Dashboard name, if no Panel Group exists above Panel).
-  For example,
-
-  * `go_to_system_flavorspage()` method to go to Admin->System->Flavors and
-
-  * `go_to_identity_projectspage()` method to go to Identity->Projects panel.
-
-* When we need to go one level deeper, i.e. go to the specific TableTab on any panel that
-  has several tabs, three components are enough - Panel Group, Panel and Tab names. For
-  example, `go_to_compute_accessandsecurity_floatingipspage()` for navigating to
-  Project->Compute->Access & Security->Floating IPs tab. Note that one cannot navigate
-  to a Panel level if that Panel has several tabs (i.e., only terminal levels could be
-  navigated to).
-
-As you might have noticed, method name components are chosen from normalized items of
-the `CORE_PAGE_STRUCTURE` dictionary, where normalization means replacing spaces with `_`
-symbol and `&` symbol with `and`, then downcasing all symbols.
-
-Once the `go_to_*page()` method's name is parsed and the proper menu item is matched in
-a dashboard, it should return the proper Page Object. For that to happen a properly
-named class should reside in a properly named module located in the right place of the
-filesystem. More specifically and top down:
-
-#. Page Object class is located in:
-
-   * <dashboard>/<panel_group>/<panel>page.py file for non-tabbed pages
-
-   * <dashboard>/<panel_group>/<panel>/<tab>page.py file for tabbed pages
-     Values <dashboard>, <panel_group>, <panel> and <tab> are the normalized versions of
-     the items from the `CORE_PAGE_STRUCTURE` dictionary.
-
-#. Within the above module a descendant of `basepage.BaseNavigationPage` should be
-   defined, its name should have the form <Panel>Page or <Tab>Page, where <Panel> and <Tab>
-   are capitalized versions of normalized <panel> and <tab> items respectively.
-
-Reusable regions
-----------------
-
-* `TableRegion` binds to the HTML Horizon table using the `TableRegion`'s `name`
-  attribute. To bind to the proper table this attribute has to be the same as
-  the `name` attribute of a `Meta` subclass of a corresponding `tables.DataTable`
-  descendant in the Python code. `TableRegion` provides all the needed facilities for
-  solving the following table-related tasks.
-
-  * Getting a specific row from a table matched by the column name and a target
-    text within that column (use `get_row()` method) or taking all the existing
-    rows on a current table page with `rows` property.
-  * Once you have a reference to a specific row, it can either be marked with
-    `mark()` for further batch actions or split to cells (using `cells` property
-    which is dictionary representing column name as a key to cell wrapper as a
-    value).
-
-  * For interacting with actions `TableRegion` provides 2 decorators, namely
-    `@bind_table_action()` and `@bind_row_action()` which bind to the actual HTML
-    button widget and decorate the specific table methods. These methods in turn
-    should click a bound button (comes as these methods' second argument after `self`)
-    and usually return a new region which is most often bound to a modal form
-    being shown after clicking that button in real Horizon.
-
-  * Another important part of `TableRegion` are the facilities for checking the
-    properties of a paged table - `assert_definition()`, `is_next_link_available()`
-    and `is_prev_link_available()` helpers and `turn_next_page()` / `turn_prev_page()`
-    which obviously cause the next / prev table page to be shown.
-
-* when interacting with modal and non-modal forms three flavors of form wrappers
-  can be used.
-
-  * `BaseFormRegion` is used for simplest forms which are usually 'Submit' /
-    'Cancel' dialogs with no fields to be filled.
-
-  * `FormRegion` is the most used wrapper which provides interaction with the
-    fields within that form. Every field is backed by its own wrapper class, while
-    the `FormRegion` acts as a container which initializes all the field wrappers in
-    its `__init__()` method. Field mappings passed to `__init__()` could be
-
-     * either a tuple of string labels, in that case the same label is used for
-       referencing the field in test code and for binding to the HTML input (should be
-       the same as `name` attribute of that widget, could be seen in Django code defining
-       that form in Horizon)
-
-     * or a dictionary, where the key will be used for referencing the test field
-       and the value will be used for binding to the HTML input. Also it is feasible
-       to provide values other than strings in that dictionary - in this case they are
-       meant to be a Python class. This Python class will be initialized as any
-       BaseRegion is usually initialized and then the value's key will be used for
-       referencing this object. This is useful when dealing with non-standard widgets
-       in forms (like Membership widget in Create/​Edit Project form or Networks widget
-       in Launch Instance form).
-
-  * `TabbedFormRegion` is a slight variation of `FormRegion`, it has several tabs
-    and thus can accept a tuple of tuples / dictionaries of field mappings, where
-    every tuple corresponds to a tab of a real form, binding order is that first
-    tuple binds to leftmost tab, which has index 0. Passing `default_tab` other than
-    0 to `TabbedFormRegion.__init__` we can make the test form to be created with
-    the tab other than the leftmost being shown immediately. Finally the method `switch_to`
-    allows us to switch to any existing form's tab.
-
-* `MessageRegion` is a small region, but is very important for asserting that
-  everything goes well in Horizon under test. Technically, the `find_message_and_dismiss`
-  method belongs to `BasePage` class, but whenever it is called, `regions.messages`
-  module is imported as well to pass a `messages.SUCCESS` / `messages.ERROR`
-  argument into. The method returns `True` / `False` depending on if the specified
-  message was found and dismissed (which could be then asserted for).
-
-Customizing tests to a specific gate environment
-------------------------------------------------
-
-* Upstream gate environment is not the only possible environment where Horizon
-  integration tests can be run. Various downstream distributions may also
-  want to run them. To ease the adoption of upstream tests to possibly
-  different conditions of a downstream gate environment, integration tests use
-  a configuration machinery backed by oslo.config library. It includes the
-  following pieces of knowledge:
-
-  * integration_tests/config.py file where all possible setting groups and
-    settings are defined along with their descriptions and defaults. If you are
-    going to add a new setting to Horizon integration tests, you should add it
-    first to this file.
-
-  * integration_tests/horizon.conf file - where all the overrides are
-    actually located. For clarity its contents mirrors the default values
-    in config.py (although technically they could be completely commented out
-    with the same result).
-
-  * To make developers' lives easier a local-only (not tracked by git)
-    counterpart of horizon.conf could exist at the same directory, named
-    'local-horizon.conf'. It is meant solely for overriding values from
-    horizon.conf that a developer's environment might differ from the gate
-    environment (like Horizon url or admin user password).
-
-* When integration tests are run by openstack-infra/devstack-gate scripts they
-  use 2 hooks to alter the devstack gate environment, namely pre_test_hook and
-  post_test_hook. Contents of both hooks are defined inside the corresponding
-  shell scripts located at 'tools/gate/integration' at the top-level of horizon
-  repo. If you find yourself asking which of the hooks you need to modify - pre
-  or post, keep the following things in mind.
-
-  * Pre hook is executed before the Devstack is deployed, that essentially
-    means that almost none of packages that are installed as OpenStack services
-    dependencies during Devstack deployment are going to be present in the
-    system. Yet all the repositories contained with `PROJECTS` variable defined
-    in `devstack-vm-gate-wrap.sh`_ script will be already cloned by the moment
-    pre hook is executed. So the natural use for it is to customize some Horizon
-    settings before they are used in operations like compressing statics etc.
-    That is how it is actually used now: it sets settings variable
-    `INTEGRATION_TESTS_SUPPORT` to `True`, so all the support code for integration
-    tests is included into Horizon static assets. If this variable was set to
-    `False`, we would not be able to run integration tests.
-
-  * Post hook is executed after Devstack is deployed, so integration tests
-    themselves are run inside that hook, as well as various test artifacts
-    collection. When you modify it, do not forget to save the exit code of
-    a tox integration tests run and emit at the end of the script - or you may
-    lose the SUCCESS/FAILURE status of the whole tests suite and tamper with the
-    job results!
-
-.. _devstack-vm-gate-wrap.sh: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh
-
-
-Writing integration tests for Horizon plugins
----------------------------------------------
-
-First, for more details on writing a Horizon plugin please refer to
-:doc:`../tutorials/plugin`. Second, there are 2 possible setups when running
-integration tests for Horizon plugins.
-
-The first setup, which is suggested to be used in gate of \*-dashboard plugins
-is to get horizon as a dependency of a plugin and then run integration tests
-using horizon.conf config file inside the plugin repo. This way the plugin augments
-the location of Horizon built-in Page Objects with the location of its own
-Page Objects, contained within the `plugin_page_path` option and the Horizon
-built-in nav structure with its own nav structure contained within
-`plugin_page_structure`. Then the plugin integration tests are run against core
-Horizon augmented with just this particular plugin content.
-
-The second setup may be used when it is needed to run integration tests for
-Horizon + several plugins. In other words, content from several plugins is
-merged into core Horizon content, then the combined integration tests from core
-Horizon and all the involved plugins are run against the resulting dashboards.
-To make this possible both options `plugin_page_path` and
-`plugin_page_structure` have MultiStrOpt type. This means that they may be
-defined several times and all the specified values will be gathered in a list,
-which is iterated over when running integration tests. In this setup it's easier to
-run the tests from Horizon repo, using the horizon.conf file within it.
-
-Also keep in mind that `plugin_page_structure` needs to be a strict JSON
-string, w/o trailing commas etc.