Documentation update for get-goal-from-strategy

In this changeset, I updated the Watcher documentation to reflect
the changes that are introduced by this blueprint.

Partially Implements: blueprint get-goal-from-strategy

Change-Id: I40be39624097365220bf7d94cbe177bbf5bbe0ed
This commit is contained in:
Vincent Françoise 2016-03-31 14:11:31 +02:00
parent ff611544fb
commit f9a1b9d3ce
12 changed files with 262 additions and 92 deletions

View File

@ -155,6 +155,7 @@ by the :ref:`Watcher API <archi_watcher_api_definition>` or the
- :ref:`Action plans <action_plan_definition>`
- :ref:`Actions <action_definition>`
- :ref:`Goals <goal_definition>`
- :ref:`Strategies <strategy_definition>`
The Watcher domain being here "*optimization of some resources provided by an
OpenStack system*".
@ -196,8 +197,6 @@ Audit, the :ref:`Strategy <strategy_definition>` relies on two sets of data:
which provides information about the past of the
:ref:`Cluster <cluster_definition>`
So far, only one :ref:`Strategy <strategy_definition>` can be associated to a
given :ref:`Goal <goal_definition>` via the main Watcher configuration file.
.. _data_model:
@ -230,13 +229,15 @@ following parameters:
- A name
- A goal to achieve
- An optional strategy
.. image:: ./images/sequence_create_audit_template.png
:width: 100%
The `Watcher API`_ just makes sure that the goal exists (i.e. it is declared
in the Watcher configuration file) and stores a new audit template in the
:ref:`Watcher Database <watcher_database_definition>`.
The `Watcher API`_ makes sure that both the specified goal (mandatory) and
its associated strategy (optional) are registered inside the :ref:`Watcher
Database <watcher_database_definition>` before storing a new audit template in
the :ref:`Watcher Database <watcher_database_definition>`.
.. _sequence_diagrams_create_and_launch_audit:
@ -260,12 +261,11 @@ the Audit in the
The :ref:`Watcher Decision Engine <watcher_decision_engine_definition>` reads
the Audit parameters from the
:ref:`Watcher Database <watcher_database_definition>`. It instantiates the
appropriate :ref:`Strategy <strategy_definition>` (using entry points)
associated to the :ref:`Goal <goal_definition>` of the
:ref:`Audit <audit_definition>` (it uses the information of the Watcher
configuration file to find the mapping between the
:ref:`Goal <goal_definition>` and the :ref:`Strategy <strategy_definition>`
python class).
appropriate :ref:`strategy <strategy_definition>` (using entry points)
given both the :ref:`goal <goal_definition>` and the strategy associated to the
parent :ref:`audit template <audit_template_definition>` of the :ref:`Audit
<audit_definition>`. If no strategy is associated to the audit template, the
strategy is dynamically selected by the Decision Engine.
The :ref:`Watcher Decision Engine <watcher_decision_engine_definition>` also
builds the :ref:`Cluster Data Model <cluster_data_model_definition>`. This

View File

@ -182,8 +182,6 @@ The configuration file is organized into the following sections:
* ``[watcher_clients_auth]`` - Keystone auth configuration for clients
* ``[watcher_applier]`` - Watcher Applier module configuration
* ``[watcher_decision_engine]`` - Watcher Decision Engine module configuration
* ``[watcher_goals]`` - Goals mapping configuration
* ``[watcher_strategies]`` - Strategy configuration
* ``[oslo_messaging_rabbit]`` - Oslo Messaging RabbitMQ driver configuration
* ``[ceilometer_client]`` - Ceilometer client configuration
* ``[cinder_client]`` - Cinder client configuration

View File

@ -56,18 +56,39 @@ watcher binary without options.
How do I run an audit of my cluster ?
-------------------------------------
First, you need to create an :ref:`audit template <audit_template_definition>`.
An :ref:`audit template <audit_template_definition>` defines an optimization
:ref:`goal <goal_definition>` to achieve (i.e. the settings of your audit).
This goal should be declared in the Watcher service configuration file
**/etc/watcher/watcher.conf**.
First, you need to find the :ref:`goal <goal_definition>` you want to achieve:
.. code:: bash
$ watcher audit-template-create my_first_audit DUMMY
$ watcher goal-list
If you get "*You must provide a username via either --os-username or via
env[OS_USERNAME]*" you may have to verify your credentials.
.. note::
If you get "*You must provide a username via either --os-username or via
env[OS_USERNAME]*" you may have to verify your credentials.
Then, you can create an :ref:`audit template <audit_template_definition>`.
An :ref:`audit template <audit_template_definition>` defines an optimization
:ref:`goal <goal_definition>` to achieve (i.e. the settings of your audit).
.. code:: bash
$ watcher audit-template-create my_first_audit_template <your_goal_uuid>
Although optional, you may want to actually set a specific strategy for your
audit template. If so, you may can search of its UUID using the following
command:
.. code:: bash
$ watcher strategy-list --goal-uuid <your_goal_uuid>
The command to create your audit template would then be:
.. code:: bash
$ watcher audit-template-create my_first_audit_template <your_goal_uuid> \
--strategy-uuid <your_strategy_uuid>
Then, you can create an audit. An audit is a request for optimizing your
cluster depending on the specified :ref:`goal <goal_definition>`.

View File

@ -15,7 +15,9 @@ plugin interface which gives anyone the ability to integrate an external
strategy in order to make use of placement algorithms.
This section gives some guidelines on how to implement and integrate custom
strategies with Watcher.
strategies with Watcher. If you wish to create a third-party package for your
plugin, you can refer to our :ref:`documentation for third-party package
creation <plugin-base_setup>`.
Pre-requisites
@ -26,64 +28,173 @@ configured so that it would provide you all the metrics you need to be able to
use your strategy.
Creating a new plugin
=====================
Create a new plugin
===================
First of all you have to:
In order to create a new strategy, you have to:
- Extend :py:class:`~.BaseStrategy`
- Implement its :py:meth:`~.BaseStrategy.execute` method
- Extend the :py:class:`~.UnclassifiedStrategy` class
- Implement its :py:meth:`~.BaseStrategy.get_name` class method to return the
**unique** ID of the new strategy you want to create. This unique ID should
be the same as the name of :ref:`the entry point we will declare later on
<strategy_plugin_add_entrypoint>`.
- Implement its :py:meth:`~.BaseStrategy.get_display_name` class method to
return the translated display name of the strategy you want to create.
Note: Do not use a variable to return the translated string so it can be
automatically collected by the translation tool.
- Implement its :py:meth:`~.BaseStrategy.get_translatable_display_name`
class method to return the translation key (actually the english display
name) of your new strategy. The value return should be the same as the
string translated in :py:meth:`~.BaseStrategy.get_display_name`.
- Implement its :py:meth:`~.BaseStrategy.execute` method to return the
solution you computed within your strategy.
Here is an example showing how you can write a plugin called ``DummyStrategy``:
Here is an example showing how you can write a plugin called ``NewStrategy``:
.. code-block:: python
import uuid
import abc
class DummyStrategy(BaseStrategy):
import six
DEFAULT_NAME = "dummy"
DEFAULT_DESCRIPTION = "Dummy Strategy"
from watcher._i18n import _
from watcher.decision_engine.strategy.strategies import base
def __init__(self, name=DEFAULT_NAME, description=DEFAULT_DESCRIPTION):
super(DummyStrategy, self).__init__(name, description)
def execute(self, model):
migration_type = 'live'
src_hypervisor = 'compute-host-1'
dst_hypervisor = 'compute-host-2'
instance_id = uuid.uuid4()
parameters = {'migration_type': migration_type,
'src_hypervisor': src_hypervisor,
'dst_hypervisor': dst_hypervisor}
self.solution.add_action(action_type="migration",
resource_id=instance_id,
class NewStrategy(base.UnclassifiedStrategy):
def __init__(self, osc=None):
super(NewStrategy, self).__init__(osc)
def execute(self, original_model):
self.solution.add_action(action_type="nop",
input_parameters=parameters)
# Do some more stuff here ...
return self.solution
@classmethod
def get_name(cls):
return "new_strategy"
@classmethod
def get_display_name(cls):
return _("New strategy")
@classmethod
def get_translatable_display_name(cls):
return "New strategy"
As you can see in the above example, the :py:meth:`~.BaseStrategy.execute`
method returns a :py:class:`~.BaseSolution` instance as required. This solution
is what wraps the abstract set of actions the strategy recommends to you. This
solution is then processed by a :ref:`planner <planner_definition>` to produce
an action plan which shall contain the sequenced flow of actions to be
an action plan which contains the sequenced flow of actions to be
executed by the :ref:`Watcher Applier <watcher_applier_definition>`.
Please note that your strategy class will be instantiated without any
parameter. Therefore, you should make sure not to make any of them required in
your ``__init__`` method.
Please note that your strategy class will expect to find the same constructor
signature as BaseStrategy to instantiate you strategy. Therefore, you should
ensure that your ``__init__`` signature is identical to the
:py:class:`~.BaseStrategy` one.
Create a new goal
=================
As stated before, the ``NewStrategy`` class extends a class called
:py:class:`~.UnclassifiedStrategy`. This class actually implements a set of
abstract methods which are defined within the :py:class:`~.BaseStrategy` parent
class.
Once you are confident in your strategy plugin, the next step is now to
classify your goal by assigning it a proper goal. To do so, you can either
reuse existing goals defined in Watcher. As of now, four goal-oriented abstract
classes are defined in Watcher:
- :py:class:`~.UnclassifiedStrategy` which is the one I mentioned up until now.
- :py:class:`~.DummyBaseStrategy` which is used by :py:class:`~.DummyStrategy`
for testing purposes.
- :py:class:`~.ServerConsolidationBaseStrategy`
- :py:class:`~.ThermalOptimizationBaseStrategy`
If none of the above actually correspond to the goal your new strategy
achieves, you can define a brand new one. To do so, you need to:
- Extend the :py:class:`~.BaseStrategy` class to make your new goal-oriented
strategy abstract class :
- Implement its :py:meth:`~.BaseStrategy.get_goal_name` class method to
return the **unique** ID of the goal you want to achieve.
- Implement its :py:meth:`~.BaseStrategy.get_goal_display_name` class method
to return the translated display name of the goal you want to achieve.
Note: Do not use a variable to return the translated string so it can be
automatically collected by the translation tool.
- Implement its :py:meth:`~.BaseStrategy.get_translatable_goal_display_name`
class method to return the goal translation key (actually the english
display name). The value return should be the same as the string translated
in :py:meth:`~.BaseStrategy.get_goal_display_name`.
Here is an example showing how you can define a new ``NEW_GOAL`` goal and
modify your ``NewStrategy`` plugin so it now achieves the latter:
.. code-block:: python
import abc
import six
from watcher._i18n import _
from watcher.decision_engine.strategy.strategies import base
@six.add_metaclass(abc.ABCMeta)
class NewGoalBaseStrategy(base.BaseStrategy):
@classmethod
def get_goal_name(cls):
return "NEW_GOAL"
@classmethod
def get_goal_display_name(cls):
return _("New goal")
@classmethod
def get_translatable_goal_display_name(cls):
return "New goal"
class NewStrategy(NewGoalBaseStrategy):
def __init__(self, osc=None):
super(NewStrategy, self).__init__(osc)
def execute(self, original_model):
self.solution.add_action(action_type="nop",
input_parameters=parameters)
# Do some more stuff here ...
return self.solution
@classmethod
def get_name(cls):
return "new_strategy"
@classmethod
def get_display_name(cls):
return _("New strategy")
@classmethod
def get_translatable_display_name(cls):
return "New strategy"
Abstract Plugin Class
=====================
Here below is the abstract :py:class:`~.BaseStrategy` class that every single
strategy should implement:
Here below is the abstract :py:class:`~.BaseStrategy` class:
.. autoclass:: watcher.decision_engine.strategy.strategies.base.BaseStrategy
:members:
:noindex:
.. _strategy_plugin_add_entrypoint:
Add a new entry point
=====================
@ -93,7 +204,9 @@ strategy must be registered as a named entry point under the
``watcher_strategies`` entry point of your ``setup.py`` file. If you are using
pbr_, this entry point should be placed in your ``setup.cfg`` file.
The name you give to your entry point has to be unique.
The name you give to your entry point has to be unique and should be the same
as the value returned by the :py:meth:`~.BaseStrategy.get_id` class method of
your strategy.
Here below is how you would proceed to register ``DummyStrategy`` using pbr_:
@ -101,7 +214,7 @@ Here below is how you would proceed to register ``DummyStrategy`` using pbr_:
[entry_points]
watcher_strategies =
dummy = thirdparty.dummy:DummyStrategy
dummy_strategy = thirdparty.dummy:DummyStrategy
To get a better understanding on how to implement a more advanced strategy,
@ -117,16 +230,10 @@ plugins when it is restarted. If a Python package containing a custom plugin is
installed within the same environment as Watcher, Watcher will automatically
make that plugin available for use.
At this point, Watcher will use your new strategy if you reference it in the
``goals`` under the ``[watcher_goals]`` section of your ``watcher.conf``
configuration file. For example, if you want to use a ``dummy`` strategy you
just installed, you would have to associate it to a goal like this:
.. code-block:: ini
[watcher_goals]
goals = BALANCE_LOAD:basic,MINIMIZE_ENERGY_CONSUMPTION:dummy
At this point, Watcher will scan and register inside the :ref:`Watcher Database
<watcher_database_definition>` all the strategies (alongside the goals they
should satisfy) you implemented upon restarting the :ref:`Watcher Decision
Engine <watcher_decision_engine_definition>`.
You should take care when installing strategy plugins. By their very nature,
there are no guarantees that utilizing them as is will be supported, as
@ -148,7 +255,6 @@ for various types of backends. A list of the available backends is located
here_. The Ceilosca project is a good example of how to create your own
pluggable backend.
Finally, if your strategy requires new metrics not covered by Ceilometer, you
can add them through a Ceilometer `plugin`_.
@ -191,7 +297,7 @@ Read usage metrics using the Watcher Cluster History Helper
Here below is the abstract ``BaseClusterHistory`` class of the Helper.
.. autoclass:: watcher.metrics_engine.cluster_history.api.BaseClusterHistory
.. autoclass:: watcher.metrics_engine.cluster_history.base.BaseClusterHistory
:members:
:noindex:

View File

@ -99,14 +99,14 @@ The :ref:`Cluster <cluster_definition>` may be divided in one or several
Cluster Data Model
==================
.. watcher-term:: watcher.metrics_engine.cluster_model_collector.api
.. watcher-term:: watcher.metrics_engine.cluster_model_collector.base
.. _cluster_history_definition:
Cluster History
===============
.. watcher-term:: watcher.metrics_engine.cluster_history.api
.. watcher-term:: watcher.metrics_engine.cluster_history.base
.. _controller_node_definition:
@ -223,8 +223,8 @@ measure of how much of the :ref:`Goal <goal_definition>` has been achieved in
respect with constraints and :ref:`SLAs <sla_definition>` defined by the
:ref:`Customer <customer_definition>`.
The way efficacy is evaluated will depend on the
:ref:`Goal <goal_definition>` to achieve.
The way efficacy is evaluated will depend on the :ref:`Goal <goal_definition>`
to achieve.
Of course, the efficacy will be relevant only as long as the
:ref:`Action Plan <action_plan_definition>` is relevant
@ -323,7 +323,7 @@ Solution
Strategy
========
.. watcher-term:: watcher.decision_engine.strategy.strategies.base
.. watcher-term:: watcher.api.controllers.v1.strategy
.. _watcher_applier_definition:

View File

@ -2,15 +2,21 @@
actor Administrator
Administrator -> "Watcher CLI" : watcher audit-template-create <name> <goal>
Administrator -> "Watcher CLI" : watcher audit-template-create <name> <goal> \
[--strategy-uuid <strategy>]
"Watcher CLI" -> "Watcher API" : POST audit_template(parameters)
"Watcher API" -> "Watcher API" : make sure goal exist in configuration
"Watcher API" -> "Watcher Database" : create new audit_template in database
"Watcher API" -> "Watcher Database" : Request if goal exists in database
"Watcher API" <-- "Watcher Database" : OK
"Watcher API" <-- "Watcher Database" : new audit template uuid
"Watcher CLI" <-- "Watcher API" : return new audit template URL in HTTP Location Header
Administrator <-- "Watcher CLI" : new audit template uuid
"Watcher API" -> "Watcher Database" : Request if strategy exists in database (if provided)
"Watcher API" <-- "Watcher Database" : OK
"Watcher API" -> "Watcher Database" : Create new audit_template in database
"Watcher API" <-- "Watcher Database" : New audit template UUID
"Watcher CLI" <-- "Watcher API" : Return new audit template URL in HTTP Location Header
Administrator <-- "Watcher CLI" : New audit template UUID
@enduml

View File

@ -4,8 +4,8 @@ actor Administrator
Administrator -> "Watcher CLI" : watcher action-plan-start <action_plan_uuid>
"Watcher CLI" -> "Watcher API" : PATCH action_plan(state=TRIGGERED)
"Watcher API" -> "Watcher Database" : action_plan.state=TRIGGERED
"Watcher CLI" -> "Watcher API" : PATCH action_plan(state=PENDING)
"Watcher API" -> "Watcher Database" : action_plan.state=PENDING
"Watcher CLI" <-- "Watcher API" : HTTP 200

Binary file not shown.

Before

Width:  |  Height:  |  Size: 22 KiB

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 26 KiB

After

Width:  |  Height:  |  Size: 25 KiB

View File

@ -14,6 +14,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""
A :ref:`Strategy <strategy_definition>` is an algorithm implementation which is
able to find a :ref:`Solution <solution_definition>` for a given
:ref:`Goal <goal_definition>`.
There may be several potential strategies which are able to achieve the same
:ref:`Goal <goal_definition>`. This is why it is possible to configure which
specific :ref:`Strategy <strategy_definition>` should be used for each goal.
Some strategies may provide better optimization results but may take more time
to find an optimal :ref:`Solution <solution_definition>`.
"""
from oslo_config import cfg
import pecan

View File

@ -169,6 +169,29 @@ class DummyBaseStrategy(BaseStrategy):
return "Dummy goal"
@six.add_metaclass(abc.ABCMeta)
class UnclassifiedStrategy(BaseStrategy):
"""This base class is used to ease the development of new strategies
The goal defined within this strategy can be used to simplify the
documentation explaining how to implement a new strategy plugin by
ommitting the need for the strategy developer to define a goal straight
away.
"""
@classmethod
def get_goal_name(cls):
return "UNCLASSIFIED"
@classmethod
def get_goal_display_name(cls):
return _("Unclassified")
@classmethod
def get_translatable_goal_display_name(cls):
return "Unclassified"
@six.add_metaclass(abc.ABCMeta)
class ServerConsolidationBaseStrategy(BaseStrategy):

View File

@ -46,10 +46,11 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy):
This strategy produces a solution resulting in more efficient
utilization of cluster resources using following four phases:
* Offload phase - handling over-utilized resources
* Consolidation phase - handling under-utilized resources
* Solution optimization - reducing number of migrations
* Deactivation of unused hypervisors
* Offload phase - handling over-utilized resources
* Consolidation phase - handling under-utilized resources
* Solution optimization - reducing number of migrations
* Deactivation of unused hypervisors
A capacity coefficients (cc) might be used to adjust optimization
thresholds. Different resources may require different coefficient
@ -58,7 +59,7 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy):
If the cc equals 1 the full resource capacity may be used, cc
values lower than 1 will lead to resource under utilization and
values higher than 1 will lead to resource overbooking.
e.g. If targeted utilization is 80% of hypervisor capacity,
e.g. If targeted utilization is 80 percent of hypervisor capacity,
the coefficient in the consolidation phase will be 0.8, but
may any lower value in the offloading phase. The lower it gets
the cluster will appear more released (distributed) for the
@ -397,12 +398,13 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy):
This is done by eliminating unnecessary or circular set of migrations
which can be replaced by a more efficient solution.
e.g.:
* A->B, B->C => replace migrations A->B, B->C with
a single migration A->C as both solution result in
VM running on hypervisor C which can be achieved with
one migration instead of two.
* A->B, B->A => remove A->B and B->A as they do not result
in a new VM placement.
* A->B, B->C => replace migrations A->B, B->C with
a single migration A->C as both solution result in
VM running on hypervisor C which can be achieved with
one migration instead of two.
* A->B, B->A => remove A->B and B->A as they do not result
in a new VM placement.
:param model: model_root object
"""
@ -502,10 +504,11 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy):
This strategy produces a solution resulting in more
efficient utilization of cluster resources using following
four phases:
* Offload phase - handling over-utilized resources
* Consolidation phase - handling under-utilized resources
* Solution optimization - reducing number of migrations
* Deactivation of unused hypervisors
* Offload phase - handling over-utilized resources
* Consolidation phase - handling under-utilized resources
* Solution optimization - reducing number of migrations
* Deactivation of unused hypervisors
:param original_model: root_model object
"""