diff --git a/doc/source/architecture.rst b/doc/source/architecture.rst index df7cfa94e..a3300b242 100644 --- a/doc/source/architecture.rst +++ b/doc/source/architecture.rst @@ -155,6 +155,7 @@ by the :ref:`Watcher API ` or the - :ref:`Action plans ` - :ref:`Actions ` - :ref:`Goals ` +- :ref:`Strategies ` The Watcher domain being here "*optimization of some resources provided by an OpenStack system*". @@ -196,8 +197,6 @@ Audit, the :ref:`Strategy ` relies on two sets of data: which provides information about the past of the :ref:`Cluster ` -So far, only one :ref:`Strategy ` can be associated to a -given :ref:`Goal ` via the main Watcher configuration file. .. _data_model: @@ -230,13 +229,15 @@ following parameters: - A name - A goal to achieve +- An optional strategy .. image:: ./images/sequence_create_audit_template.png :width: 100% -The `Watcher API`_ just makes sure that the goal exists (i.e. it is declared -in the Watcher configuration file) and stores a new audit template in the -:ref:`Watcher Database `. +The `Watcher API`_ makes sure that both the specified goal (mandatory) and +its associated strategy (optional) are registered inside the :ref:`Watcher +Database ` before storing a new audit template in +the :ref:`Watcher Database `. .. _sequence_diagrams_create_and_launch_audit: @@ -260,12 +261,11 @@ the Audit in the The :ref:`Watcher Decision Engine ` reads the Audit parameters from the :ref:`Watcher Database `. It instantiates the -appropriate :ref:`Strategy ` (using entry points) -associated to the :ref:`Goal ` of the -:ref:`Audit ` (it uses the information of the Watcher -configuration file to find the mapping between the -:ref:`Goal ` and the :ref:`Strategy ` -python class). +appropriate :ref:`strategy ` (using entry points) +given both the :ref:`goal ` and the strategy associated to the +parent :ref:`audit template ` of the :ref:`Audit +`. If no strategy is associated to the audit template, the +strategy is dynamically selected by the Decision Engine. The :ref:`Watcher Decision Engine ` also builds the :ref:`Cluster Data Model `. This diff --git a/doc/source/deploy/configuration.rst b/doc/source/deploy/configuration.rst index 099814f8c..2adcc643f 100644 --- a/doc/source/deploy/configuration.rst +++ b/doc/source/deploy/configuration.rst @@ -182,8 +182,6 @@ The configuration file is organized into the following sections: * ``[watcher_clients_auth]`` - Keystone auth configuration for clients * ``[watcher_applier]`` - Watcher Applier module configuration * ``[watcher_decision_engine]`` - Watcher Decision Engine module configuration -* ``[watcher_goals]`` - Goals mapping configuration -* ``[watcher_strategies]`` - Strategy configuration * ``[oslo_messaging_rabbit]`` - Oslo Messaging RabbitMQ driver configuration * ``[ceilometer_client]`` - Ceilometer client configuration * ``[cinder_client]`` - Cinder client configuration diff --git a/doc/source/deploy/user-guide.rst b/doc/source/deploy/user-guide.rst index fcd2c6006..d74e16850 100644 --- a/doc/source/deploy/user-guide.rst +++ b/doc/source/deploy/user-guide.rst @@ -56,18 +56,39 @@ watcher binary without options. How do I run an audit of my cluster ? ------------------------------------- -First, you need to create an :ref:`audit template `. -An :ref:`audit template ` defines an optimization -:ref:`goal ` to achieve (i.e. the settings of your audit). -This goal should be declared in the Watcher service configuration file -**/etc/watcher/watcher.conf**. +First, you need to find the :ref:`goal ` you want to achieve: .. code:: bash - $ watcher audit-template-create my_first_audit DUMMY + $ watcher goal-list -If you get "*You must provide a username via either --os-username or via -env[OS_USERNAME]*" you may have to verify your credentials. +.. note:: + + If you get "*You must provide a username via either --os-username or via + env[OS_USERNAME]*" you may have to verify your credentials. + +Then, you can create an :ref:`audit template `. +An :ref:`audit template ` defines an optimization +:ref:`goal ` to achieve (i.e. the settings of your audit). + +.. code:: bash + + $ watcher audit-template-create my_first_audit_template + +Although optional, you may want to actually set a specific strategy for your +audit template. If so, you may can search of its UUID using the following +command: + +.. code:: bash + + $ watcher strategy-list --goal-uuid + +The command to create your audit template would then be: + +.. code:: bash + + $ watcher audit-template-create my_first_audit_template \ + --strategy-uuid Then, you can create an audit. An audit is a request for optimizing your cluster depending on the specified :ref:`goal `. diff --git a/doc/source/dev/plugin/strategy-plugin.rst b/doc/source/dev/plugin/strategy-plugin.rst index a83b5b78b..6598bab42 100644 --- a/doc/source/dev/plugin/strategy-plugin.rst +++ b/doc/source/dev/plugin/strategy-plugin.rst @@ -15,7 +15,9 @@ plugin interface which gives anyone the ability to integrate an external strategy in order to make use of placement algorithms. This section gives some guidelines on how to implement and integrate custom -strategies with Watcher. +strategies with Watcher. If you wish to create a third-party package for your +plugin, you can refer to our :ref:`documentation for third-party package +creation `. Pre-requisites @@ -26,64 +28,173 @@ configured so that it would provide you all the metrics you need to be able to use your strategy. -Creating a new plugin -===================== +Create a new plugin +=================== -First of all you have to: +In order to create a new strategy, you have to: -- Extend :py:class:`~.BaseStrategy` -- Implement its :py:meth:`~.BaseStrategy.execute` method +- Extend the :py:class:`~.UnclassifiedStrategy` class +- Implement its :py:meth:`~.BaseStrategy.get_name` class method to return the + **unique** ID of the new strategy you want to create. This unique ID should + be the same as the name of :ref:`the entry point we will declare later on + `. +- Implement its :py:meth:`~.BaseStrategy.get_display_name` class method to + return the translated display name of the strategy you want to create. + Note: Do not use a variable to return the translated string so it can be + automatically collected by the translation tool. +- Implement its :py:meth:`~.BaseStrategy.get_translatable_display_name` + class method to return the translation key (actually the english display + name) of your new strategy. The value return should be the same as the + string translated in :py:meth:`~.BaseStrategy.get_display_name`. +- Implement its :py:meth:`~.BaseStrategy.execute` method to return the + solution you computed within your strategy. -Here is an example showing how you can write a plugin called ``DummyStrategy``: +Here is an example showing how you can write a plugin called ``NewStrategy``: .. code-block:: python - import uuid + import abc - class DummyStrategy(BaseStrategy): + import six - DEFAULT_NAME = "dummy" - DEFAULT_DESCRIPTION = "Dummy Strategy" + from watcher._i18n import _ + from watcher.decision_engine.strategy.strategies import base - def __init__(self, name=DEFAULT_NAME, description=DEFAULT_DESCRIPTION): - super(DummyStrategy, self).__init__(name, description) - def execute(self, model): - migration_type = 'live' - src_hypervisor = 'compute-host-1' - dst_hypervisor = 'compute-host-2' - instance_id = uuid.uuid4() - parameters = {'migration_type': migration_type, - 'src_hypervisor': src_hypervisor, - 'dst_hypervisor': dst_hypervisor} - self.solution.add_action(action_type="migration", - resource_id=instance_id, + class NewStrategy(base.UnclassifiedStrategy): + + def __init__(self, osc=None): + super(NewStrategy, self).__init__(osc) + + def execute(self, original_model): + self.solution.add_action(action_type="nop", input_parameters=parameters) # Do some more stuff here ... return self.solution + @classmethod + def get_name(cls): + return "new_strategy" + + @classmethod + def get_display_name(cls): + return _("New strategy") + + @classmethod + def get_translatable_display_name(cls): + return "New strategy" + + As you can see in the above example, the :py:meth:`~.BaseStrategy.execute` method returns a :py:class:`~.BaseSolution` instance as required. This solution is what wraps the abstract set of actions the strategy recommends to you. This solution is then processed by a :ref:`planner ` to produce -an action plan which shall contain the sequenced flow of actions to be +an action plan which contains the sequenced flow of actions to be executed by the :ref:`Watcher Applier `. -Please note that your strategy class will be instantiated without any -parameter. Therefore, you should make sure not to make any of them required in -your ``__init__`` method. +Please note that your strategy class will expect to find the same constructor +signature as BaseStrategy to instantiate you strategy. Therefore, you should +ensure that your ``__init__`` signature is identical to the +:py:class:`~.BaseStrategy` one. + + +Create a new goal +================= + +As stated before, the ``NewStrategy`` class extends a class called +:py:class:`~.UnclassifiedStrategy`. This class actually implements a set of +abstract methods which are defined within the :py:class:`~.BaseStrategy` parent +class. + +Once you are confident in your strategy plugin, the next step is now to +classify your goal by assigning it a proper goal. To do so, you can either +reuse existing goals defined in Watcher. As of now, four goal-oriented abstract +classes are defined in Watcher: + +- :py:class:`~.UnclassifiedStrategy` which is the one I mentioned up until now. +- :py:class:`~.DummyBaseStrategy` which is used by :py:class:`~.DummyStrategy` + for testing purposes. +- :py:class:`~.ServerConsolidationBaseStrategy` +- :py:class:`~.ThermalOptimizationBaseStrategy` + +If none of the above actually correspond to the goal your new strategy +achieves, you can define a brand new one. To do so, you need to: + +- Extend the :py:class:`~.BaseStrategy` class to make your new goal-oriented + strategy abstract class : +- Implement its :py:meth:`~.BaseStrategy.get_goal_name` class method to + return the **unique** ID of the goal you want to achieve. +- Implement its :py:meth:`~.BaseStrategy.get_goal_display_name` class method + to return the translated display name of the goal you want to achieve. + Note: Do not use a variable to return the translated string so it can be + automatically collected by the translation tool. +- Implement its :py:meth:`~.BaseStrategy.get_translatable_goal_display_name` + class method to return the goal translation key (actually the english + display name). The value return should be the same as the string translated + in :py:meth:`~.BaseStrategy.get_goal_display_name`. + +Here is an example showing how you can define a new ``NEW_GOAL`` goal and +modify your ``NewStrategy`` plugin so it now achieves the latter: + +.. code-block:: python + + import abc + + import six + + from watcher._i18n import _ + from watcher.decision_engine.strategy.strategies import base + + @six.add_metaclass(abc.ABCMeta) + class NewGoalBaseStrategy(base.BaseStrategy): + + @classmethod + def get_goal_name(cls): + return "NEW_GOAL" + + @classmethod + def get_goal_display_name(cls): + return _("New goal") + + @classmethod + def get_translatable_goal_display_name(cls): + return "New goal" + + + class NewStrategy(NewGoalBaseStrategy): + + def __init__(self, osc=None): + super(NewStrategy, self).__init__(osc) + + def execute(self, original_model): + self.solution.add_action(action_type="nop", + input_parameters=parameters) + # Do some more stuff here ... + return self.solution + + @classmethod + def get_name(cls): + return "new_strategy" + + @classmethod + def get_display_name(cls): + return _("New strategy") + + @classmethod + def get_translatable_display_name(cls): + return "New strategy" Abstract Plugin Class ===================== -Here below is the abstract :py:class:`~.BaseStrategy` class that every single -strategy should implement: +Here below is the abstract :py:class:`~.BaseStrategy` class: .. autoclass:: watcher.decision_engine.strategy.strategies.base.BaseStrategy :members: :noindex: +.. _strategy_plugin_add_entrypoint: Add a new entry point ===================== @@ -93,7 +204,9 @@ strategy must be registered as a named entry point under the ``watcher_strategies`` entry point of your ``setup.py`` file. If you are using pbr_, this entry point should be placed in your ``setup.cfg`` file. -The name you give to your entry point has to be unique. +The name you give to your entry point has to be unique and should be the same +as the value returned by the :py:meth:`~.BaseStrategy.get_id` class method of +your strategy. Here below is how you would proceed to register ``DummyStrategy`` using pbr_: @@ -101,7 +214,7 @@ Here below is how you would proceed to register ``DummyStrategy`` using pbr_: [entry_points] watcher_strategies = - dummy = thirdparty.dummy:DummyStrategy + dummy_strategy = thirdparty.dummy:DummyStrategy To get a better understanding on how to implement a more advanced strategy, @@ -117,16 +230,10 @@ plugins when it is restarted. If a Python package containing a custom plugin is installed within the same environment as Watcher, Watcher will automatically make that plugin available for use. -At this point, Watcher will use your new strategy if you reference it in the -``goals`` under the ``[watcher_goals]`` section of your ``watcher.conf`` -configuration file. For example, if you want to use a ``dummy`` strategy you -just installed, you would have to associate it to a goal like this: - -.. code-block:: ini - - [watcher_goals] - goals = BALANCE_LOAD:basic,MINIMIZE_ENERGY_CONSUMPTION:dummy - +At this point, Watcher will scan and register inside the :ref:`Watcher Database +` all the strategies (alongside the goals they +should satisfy) you implemented upon restarting the :ref:`Watcher Decision +Engine `. You should take care when installing strategy plugins. By their very nature, there are no guarantees that utilizing them as is will be supported, as @@ -148,7 +255,6 @@ for various types of backends. A list of the available backends is located here_. The Ceilosca project is a good example of how to create your own pluggable backend. - Finally, if your strategy requires new metrics not covered by Ceilometer, you can add them through a Ceilometer `plugin`_. @@ -191,7 +297,7 @@ Read usage metrics using the Watcher Cluster History Helper Here below is the abstract ``BaseClusterHistory`` class of the Helper. -.. autoclass:: watcher.metrics_engine.cluster_history.api.BaseClusterHistory +.. autoclass:: watcher.metrics_engine.cluster_history.base.BaseClusterHistory :members: :noindex: diff --git a/doc/source/glossary.rst b/doc/source/glossary.rst index 84da166e0..d3ea28738 100644 --- a/doc/source/glossary.rst +++ b/doc/source/glossary.rst @@ -99,14 +99,14 @@ The :ref:`Cluster ` may be divided in one or several Cluster Data Model ================== -.. watcher-term:: watcher.metrics_engine.cluster_model_collector.api +.. watcher-term:: watcher.metrics_engine.cluster_model_collector.base .. _cluster_history_definition: Cluster History =============== -.. watcher-term:: watcher.metrics_engine.cluster_history.api +.. watcher-term:: watcher.metrics_engine.cluster_history.base .. _controller_node_definition: @@ -223,8 +223,8 @@ measure of how much of the :ref:`Goal ` has been achieved in respect with constraints and :ref:`SLAs ` defined by the :ref:`Customer `. -The way efficacy is evaluated will depend on the -:ref:`Goal ` to achieve. +The way efficacy is evaluated will depend on the :ref:`Goal ` +to achieve. Of course, the efficacy will be relevant only as long as the :ref:`Action Plan ` is relevant @@ -323,7 +323,7 @@ Solution Strategy ======== -.. watcher-term:: watcher.decision_engine.strategy.strategies.base +.. watcher-term:: watcher.api.controllers.v1.strategy .. _watcher_applier_definition: diff --git a/doc/source/image_src/plantuml/sequence_create_audit_template.txt b/doc/source/image_src/plantuml/sequence_create_audit_template.txt index 98f8dc8cd..455cb3a77 100644 --- a/doc/source/image_src/plantuml/sequence_create_audit_template.txt +++ b/doc/source/image_src/plantuml/sequence_create_audit_template.txt @@ -2,15 +2,21 @@ actor Administrator -Administrator -> "Watcher CLI" : watcher audit-template-create +Administrator -> "Watcher CLI" : watcher audit-template-create \ +[--strategy-uuid ] "Watcher CLI" -> "Watcher API" : POST audit_template(parameters) -"Watcher API" -> "Watcher API" : make sure goal exist in configuration -"Watcher API" -> "Watcher Database" : create new audit_template in database +"Watcher API" -> "Watcher Database" : Request if goal exists in database +"Watcher API" <-- "Watcher Database" : OK -"Watcher API" <-- "Watcher Database" : new audit template uuid -"Watcher CLI" <-- "Watcher API" : return new audit template URL in HTTP Location Header -Administrator <-- "Watcher CLI" : new audit template uuid +"Watcher API" -> "Watcher Database" : Request if strategy exists in database (if provided) +"Watcher API" <-- "Watcher Database" : OK + +"Watcher API" -> "Watcher Database" : Create new audit_template in database +"Watcher API" <-- "Watcher Database" : New audit template UUID + +"Watcher CLI" <-- "Watcher API" : Return new audit template URL in HTTP Location Header +Administrator <-- "Watcher CLI" : New audit template UUID @enduml diff --git a/doc/source/image_src/plantuml/sequence_launch_action_plan.txt b/doc/source/image_src/plantuml/sequence_launch_action_plan.txt index 92469e654..88785ced5 100644 --- a/doc/source/image_src/plantuml/sequence_launch_action_plan.txt +++ b/doc/source/image_src/plantuml/sequence_launch_action_plan.txt @@ -4,8 +4,8 @@ actor Administrator Administrator -> "Watcher CLI" : watcher action-plan-start -"Watcher CLI" -> "Watcher API" : PATCH action_plan(state=TRIGGERED) -"Watcher API" -> "Watcher Database" : action_plan.state=TRIGGERED +"Watcher CLI" -> "Watcher API" : PATCH action_plan(state=PENDING) +"Watcher API" -> "Watcher Database" : action_plan.state=PENDING "Watcher CLI" <-- "Watcher API" : HTTP 200 diff --git a/doc/source/images/sequence_create_audit_template.png b/doc/source/images/sequence_create_audit_template.png index 57c539aec..92b57bc54 100644 Binary files a/doc/source/images/sequence_create_audit_template.png and b/doc/source/images/sequence_create_audit_template.png differ diff --git a/doc/source/images/sequence_launch_action_plan.png b/doc/source/images/sequence_launch_action_plan.png index 2c7ece74e..97465c568 100644 Binary files a/doc/source/images/sequence_launch_action_plan.png and b/doc/source/images/sequence_launch_action_plan.png differ diff --git a/watcher/api/controllers/v1/strategy.py b/watcher/api/controllers/v1/strategy.py index 4dcc6e59d..d0889a119 100644 --- a/watcher/api/controllers/v1/strategy.py +++ b/watcher/api/controllers/v1/strategy.py @@ -14,6 +14,19 @@ # See the License for the specific language governing permissions and # limitations under the License. +""" +A :ref:`Strategy ` is an algorithm implementation which is +able to find a :ref:`Solution ` for a given +:ref:`Goal `. + +There may be several potential strategies which are able to achieve the same +:ref:`Goal `. This is why it is possible to configure which +specific :ref:`Strategy ` should be used for each goal. + +Some strategies may provide better optimization results but may take more time +to find an optimal :ref:`Solution `. +""" + from oslo_config import cfg import pecan diff --git a/watcher/decision_engine/strategy/strategies/base.py b/watcher/decision_engine/strategy/strategies/base.py index 8bcb99346..e35b92d18 100644 --- a/watcher/decision_engine/strategy/strategies/base.py +++ b/watcher/decision_engine/strategy/strategies/base.py @@ -169,6 +169,29 @@ class DummyBaseStrategy(BaseStrategy): return "Dummy goal" +@six.add_metaclass(abc.ABCMeta) +class UnclassifiedStrategy(BaseStrategy): + """This base class is used to ease the development of new strategies + + The goal defined within this strategy can be used to simplify the + documentation explaining how to implement a new strategy plugin by + ommitting the need for the strategy developer to define a goal straight + away. + """ + + @classmethod + def get_goal_name(cls): + return "UNCLASSIFIED" + + @classmethod + def get_goal_display_name(cls): + return _("Unclassified") + + @classmethod + def get_translatable_goal_display_name(cls): + return "Unclassified" + + @six.add_metaclass(abc.ABCMeta) class ServerConsolidationBaseStrategy(BaseStrategy): diff --git a/watcher/decision_engine/strategy/strategies/vm_workload_consolidation.py b/watcher/decision_engine/strategy/strategies/vm_workload_consolidation.py index 4c89dc2cc..c897e2322 100644 --- a/watcher/decision_engine/strategy/strategies/vm_workload_consolidation.py +++ b/watcher/decision_engine/strategy/strategies/vm_workload_consolidation.py @@ -46,10 +46,11 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy): This strategy produces a solution resulting in more efficient utilization of cluster resources using following four phases: - * Offload phase - handling over-utilized resources - * Consolidation phase - handling under-utilized resources - * Solution optimization - reducing number of migrations - * Deactivation of unused hypervisors + + * Offload phase - handling over-utilized resources + * Consolidation phase - handling under-utilized resources + * Solution optimization - reducing number of migrations + * Deactivation of unused hypervisors A capacity coefficients (cc) might be used to adjust optimization thresholds. Different resources may require different coefficient @@ -58,7 +59,7 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy): If the cc equals 1 the full resource capacity may be used, cc values lower than 1 will lead to resource under utilization and values higher than 1 will lead to resource overbooking. - e.g. If targeted utilization is 80% of hypervisor capacity, + e.g. If targeted utilization is 80 percent of hypervisor capacity, the coefficient in the consolidation phase will be 0.8, but may any lower value in the offloading phase. The lower it gets the cluster will appear more released (distributed) for the @@ -397,12 +398,13 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy): This is done by eliminating unnecessary or circular set of migrations which can be replaced by a more efficient solution. e.g.: - * A->B, B->C => replace migrations A->B, B->C with - a single migration A->C as both solution result in - VM running on hypervisor C which can be achieved with - one migration instead of two. - * A->B, B->A => remove A->B and B->A as they do not result - in a new VM placement. + + * A->B, B->C => replace migrations A->B, B->C with + a single migration A->C as both solution result in + VM running on hypervisor C which can be achieved with + one migration instead of two. + * A->B, B->A => remove A->B and B->A as they do not result + in a new VM placement. :param model: model_root object """ @@ -502,10 +504,11 @@ class VMWorkloadConsolidation(base.ServerConsolidationBaseStrategy): This strategy produces a solution resulting in more efficient utilization of cluster resources using following four phases: - * Offload phase - handling over-utilized resources - * Consolidation phase - handling under-utilized resources - * Solution optimization - reducing number of migrations - * Deactivation of unused hypervisors + + * Offload phase - handling over-utilized resources + * Consolidation phase - handling under-utilized resources + * Solution optimization - reducing number of migrations + * Deactivation of unused hypervisors :param original_model: root_model object """