Add the Profiler panel to the Developer dashboard

Provide both pythonic Django part and the static assets (angular
directives and styles) for the new panel.

DEPLOY NOTES:

To enable panel itself, copy
openstack_dashboard/local/local_settings.d/_9030_profiler_settings.py.example
file from the previous commit to
openstack_dashboard/local/local_settings.d/_9030_profiler_settings.py
and copy openstack_dashboard/contrib/developer/enabled/_9030_profiler.py
to openstack_dashboard/local/enabled/_9030_profiler.py

To support storing profiler data on server-side, MongoDB cluster needs
to be installed on Devstack host (default configuration), see
https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/#install-mongodb-community-edition
for instructions. Then, change net:bindIp: key to 0.0.0.0 inside
/etc/mongod.conf and invoke `sudo service mongod restart` for the
changes to have an effect.

Implements-blueprint: openstack-profiler-at-developer-dashboard
Change-Id: Ice7b8b4b4decad2c45a9edef3f3c4cc2ff759de4
This commit is contained in:
Timur Sufiev 2016-08-30 21:30:57 +03:00 committed by Timur Sufiev
parent 6da0e2d281
commit 4ceeef5376
22 changed files with 625 additions and 9 deletions

View File

@ -90,6 +90,7 @@ After You Write Your Patch
Once you've made your changes, there are a few things to do:
* Make sure the unit tests and linting tasks pass by running ``tox``
* Take a look at your patch in API profiler, i.e. how it impacts the performance. See `Profiling Pages`_.
* Make sure your code is ready for translation: See :ref:`pseudo_translation`.
* Make sure your code is up-to-date with the latest master: ``git pull --rebase``
* Finally, run ``git review`` to upload your changes to Gerrit for review.
@ -100,6 +101,34 @@ If the review is approved, it is sent to Jenkins to verify the unit tests pass
and it can be merged cleanly. Once Jenkins approves it, the change will be
merged to the master repository and it's time to celebrate!
Profiling Pages
---------------
In Ocata release of Horizon a new "OpenStack Profiler" panel is introduced within
a Developer dashboard. Once it is enabled and all prerequisites are set up, you
can see what API calls Horizon actually makes when rendering a specific page. To
re-render the page while profiling it, you'll need to use "Profile" drop-down
menu located left to the User menu in top right corner of the screen. In order to
be able to use "Profile" menu the following steps need to be done:
#. Ensure that the Developer dashboard is enabled (copy _9001_developer.py file
from the openstack_dashboard/contrib/developer/enabled folder into the
openstack_dashboard/local/enabled folder if it is not already there).
#. Copy openstack_dashboard/local/local_settings.d/_9030_profiler_settings.py.example
file to openstack_dashboard/local/local_settings.d/_9030_profiler_settings.py
#. Copy openstack_dashboard/contrib/developer/enabled/_9030_profiler.py to
openstack_dashboard/local/enabled/_9030_profiler.py .
#. To support storing profiler data on server-side, MongoDB cluster needs
to be installed on Devstack host (default configuration), see `Installing MongoDB`_.
Then, change net:bindIp: key to 0.0.0.0 inside /etc/mongod.conf and invoke
``sudo service mongod restart`` for the changes to have an effect.
#. Re-collect and re-compress static assets.
#. Re-start the production web-server in case you are serving Horizon from it.
#. The "Profile" drop-down menu should appear in the top-right corner, you are
ready to profile your pages!
.. _installing MongoDB: https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/#install-mongodb-community-edition
Etiquette
=========

View File

@ -60,6 +60,10 @@ def openstack(request):
# Adding webroot access
context['WEBROOT'] = getattr(settings, "WEBROOT", "/")
# Adding profiler support flag
enabled = getattr(settings, 'OPENSTACK_PROFILER', {}).get('enabled', False)
context['profiler_enabled'] = enabled
# Search for external plugins and append to javascript message catalog
# internal plugins are under the openstack_dashboard domain
# so we exclude them from the js_catalog

View File

@ -0,0 +1,21 @@
# Copyright 2016 Mirantis Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
PANEL = 'profiler'
PANEL_GROUP = 'default'
PANEL_DASHBOARD = 'developer'
ADD_PANEL = 'openstack_dashboard.contrib.developer.profiler.panel.Profiler'

View File

@ -29,13 +29,19 @@ from openstack_dashboard.contrib.developer.profiler import api
_REQUIRED_KEYS = ("base_id", "hmac_key")
_OPTIONAL_KEYS = ("parent_id",)
PROFILER_SETTINGS = getattr(settings, 'OPENSTACK_PROFILER', {})
PROFILER_CONF = getattr(settings, 'OPENSTACK_PROFILER', {})
PROFILER_ENABLED = PROFILER_CONF.get('enabled', False)
class ProfilerClientMiddleware(object):
def __init__(self):
if not PROFILER_ENABLED:
raise exceptions.MiddlewareNotUsed()
super(ProfilerClientMiddleware, self).__init__()
def process_request(self, request):
if 'profile_page' in request.COOKIES:
hmac_key = PROFILER_SETTINGS.get('keys')[0]
hmac_key = PROFILER_CONF.get('keys')[0]
profiler.init(hmac_key)
for hdr_key, hdr_value in web.get_trace_id_headers().items():
request.META[hdr_key] = hdr_value
@ -44,12 +50,10 @@ class ProfilerClientMiddleware(object):
class ProfilerMiddleware(object):
def __init__(self):
self.name = PROFILER_SETTINGS.get('facility_name', 'horizon')
self.hmac_keys = PROFILER_SETTINGS.get('keys')
self._enabled = PROFILER_SETTINGS.get('enabled', False)
if self._enabled:
api.init_notifier(PROFILER_SETTINGS.get(
'notifier_connection_string', 'mongodb://'))
self.name = PROFILER_CONF.get('facility_name', 'horizon')
self.hmac_keys = PROFILER_CONF.get('keys', [])
if PROFILER_ENABLED:
api.init_notifier(PROFILER_CONF.get('notifier_connection_string'))
else:
raise exceptions.MiddlewareNotUsed()

View File

@ -0,0 +1,25 @@
# Copyright 2016 Mirantis Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from django.conf import settings
from django.utils.translation import ugettext_lazy as _
import horizon
if getattr(settings, 'OPENSTACK_PROFILER', {}).get('enabled', False):
class Profiler(horizon.Panel):
name = _("OpenStack Profiler")
slug = 'profiler'

View File

@ -0,0 +1,21 @@
{% if not_list %}
<div class="dropdown" ng-controller="profilerActionsController as ctrl">
{% else %}
<li class="dropdown" ng-controller="profilerActionsController as ctrl">
{% endif %}
<a data-toggle="dropdown" href="#" class="dropdown-toggle" role="button"
aria-expanded="false">
<span class="fa fa-calculator"></span>
Profile
<span class="fa fa-caret-down"></span>
</a>
<ul class="dropdown-menu dropdown-menu-right">
<li><a href="#" ng-click="ctrl.profilePage()">
<span class="fa fa-refresh"></span> Profile Current Page
</a></li>
</ul>
{% if not_list %}
</div>
{% else %}
</li>
{% endif %}

View File

@ -0,0 +1,49 @@
{% extends 'base.html' %}
{% load i18n %}
{% block title %}
{% trans "OpenStack Profiler" %}
{% endblock %}
{% block page_header %}
<h1>OpenStack Profiler</h1>
{% endblock %}
{% block main %}
<div class="profiler" ng-cloak ng-controller="topProfilerController as ctrl">
<h2>Traces</h2>
<table st-table="ctrl.tracesDisplayed"
st-safe-src="ctrl.traces"
hz-table
class="table table-striped table-rsp table-detail">
<thead>
<tr>
<th class="expander"></th>
<th>Traced At</th>
<th>Trace ID</th>
<th>Origin</th>
<th>Request time</th>
</tr>
</thead>
<tbody>
<tr ng-repeat-start="trace in ctrl.tracesDisplayed track by trace.id">
<td class="expander">
<span class="fa fa-chevron-right" hz-expand-detail item="trace">
</span>
</td>
<td>{$ trace.timestamp $}</td>
<td>{$ trace.id $}</td>
<td>{$ trace.origin $}</td>
<td>{$ trace.request_time $}</td>
</tr>
<tr class="detail-row"
ng-repeat-end>
<td></td>
<td class="detail" colspan="4">
<table trace-table="trace"></table>
</td>
</tr>
</tbody>
</table>
</div>
{% endblock %}

View File

@ -0,0 +1,24 @@
# Copyright 2016 Mirantis Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from django.conf.urls import url
from openstack_dashboard.contrib.developer.profiler import views
urlpatterns = [
url(r'^$', views.IndexView.as_view(), name='index'),
]

View File

@ -0,0 +1,49 @@
# Copyright 2016 Mirantis Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from django.utils.translation import ugettext_lazy as _
from django.views import generic
from horizon import views
from openstack_dashboard.api.rest import urls
from openstack_dashboard.api.rest import utils
from openstack_dashboard.contrib.developer.profiler import api
class IndexView(views.HorizonTemplateView):
template_name = 'developer/profiler/index.html'
page_title = _("OpenStack Profiler")
def get_context_data(self, **kwargs):
context = super(IndexView, self).get_context_data(**kwargs)
return context
@urls.register
class Traces(generic.View):
url_regex = r'profiler/traces$'
@utils.ajax()
def get(self, request):
return api.list_traces(request)
@urls.register
class Trace(generic.View):
url_regex = r'profiler/traces/(?P<trace_id>[^/]+)/$'
@utils.ajax()
def get(self, request, trace_id):
return api.get_trace(request, trace_id)

View File

@ -26,7 +26,8 @@
angular
.module('horizon.dashboard.developer', [
'horizon.dashboard.developer.theme-preview',
'horizon.dashboard.developer.resource-browser'
'horizon.dashboard.developer.resource-browser',
'horizon.dashboard.developer.profiler'
])
.config(config);

View File

@ -1,2 +1,3 @@
// Top level file for Developer dashboard SCSS
@import "theme-preview/theme-preview";
@import "profiler/profiler";

View File

@ -0,0 +1,161 @@
/*
* (c) Copyright 2015 Mirantis Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
(function () {
'use strict';
angular
.module('horizon.dashboard.developer.profiler')
.controller('topProfilerController', topProfilerController)
.controller('sharedProfilerController', sharedProfilerController)
.controller('profilerActionsController', actionsController);
/**
* @ngdoc controller
* @name horizon.dashboard.developer.topProfilerController
* @description
* This is the top-level controller for the Profiler view.
* Its primary purpose is hand the list of traces over to profiler table widget.
*/
topProfilerController.$inject = ['horizon.framework.util.http.service'];
function topProfilerController($http) {
var ctrl = this;
$http.get('/api/profiler/traces').then(function(response) {
ctrl.traces = response.data;
ctrl.tracesDisplayed = response.data;
});
}
/**
* @ngdoc controller
* @name horizon.dashboard.developer.sharedProfilerController
* @description
* This is the controller being used inside <trace-table> directive, it is
* shared between all the trace node directives which recursively form the
* whole trace (hence the name). It contains various helper methods which
* are used while rendering trace node.
*/
sharedProfilerController.$inject = [
'$modal',
'$rootScope',
'$templateCache',
'horizon.dashboard.developer.profiler.basePath'
];
function sharedProfilerController($modal, $rootScope, $templateCache, basePath) {
var ctrl = this;
ctrl.getWidth = getWidth;
ctrl.getStarted = getStarted;
ctrl.isImportant = isImportant;
ctrl.display = display;
ctrl.toggleChildren = toggleChildren;
ctrl.getLeafCls = getLeafCls;
ctrl.getBranchCls = getBranchCls;
function toggleChildren(data) {
function rec(data, value, nonRoot) {
if (nonRoot) {
data.visible = value;
}
// don't expand nodes collapsed explicitly when expanding one of their
// parents
if (!(value && !data.childrenVisible)) {
data.children.forEach(function(child) {
rec(child, value, true);
});
}
}
data.childrenVisible = !data.childrenVisible;
rec(data, data.childrenVisible);
}
function getLeafCls(data) {
return data.is_leaf ? 'fa-cloud' : '';
}
function getBranchCls(data) {
if (!data.children.length) {
return '';
}
return data.children[0].visible ? 'fa-minus' : 'fa-plus';
}
function getWidth(data, rootData) {
var full_duration = rootData.info.finished;
var duration = (data.info.finished - data.info.started) * 100.0 / full_duration;
return (duration >= 0.5) ? duration : 0.5;
}
function getStarted(data, rootData) {
var full_duration = rootData.info.finished;
return data.info.started * 100.0 / full_duration;
}
function isImportant(data) {
return ["total", "wsgi", "rpc"].indexOf(data.info.name) != -1;
}
function display(data){
var scope = $rootScope.$new();
var info = angular.copy(data.info);
var metadata = {};
angular.forEach(info, function(value, key) {
var parts = key.split(".");
if (parts[0] == "meta") {
if (parts.length == 2){
this[parts[1]] = value;
}
else{
var group_name = parts[1];
if (!(group_name in this))
this[group_name] = {};
this[group_name][parts[2]] = value;
}
}
}, metadata);
info["duration"] = info["finished"] - info["started"];
info["metadata"] = JSON.stringify(metadata, "", 4);
scope.info = info;
scope.columns = ["name", "project", "service", "host", "started",
"finished", "duration", "metadata"];
$modal.open({
"size": "lg",
"template": $templateCache.get(basePath + 'profiler.details.html'),
"scope": scope
});
}
}
/**
* @ngdoc controller
* @name horizon.dashboard.developer.profilerActionsController
* @description
* This is the controller being used in header partial template for invoking
* various profiling actions through a drop-down control.
*/
actionsController.$inject = ['$cookies'];
function actionsController($cookies) {
var ctrl = this;
ctrl.profilePage = profilePage;
function profilePage() {
$cookies.put('profile_page', true);
window.location.reload();
}
}
})();

View File

@ -0,0 +1,17 @@
<div class="modal-header">
Trace Point Details
</div>
<div class="modal-body">
<div class="row" ng-repeat="column in columns">
<div class="col-md-2 text-right text-capitalize">
<strong>{$ column $}</strong>
</div>
<div class="col-md-10 text-left">
<pre ng-if="column == 'metadata'">{$ info[column] $}</pre>
<span ng-if="column != 'metadata'">{$ info[column] $}</span>
</div>
</div>
</div>
<div class="modal-footer">
<span class="fa fa-cloud"></span>
</div>

View File

@ -0,0 +1,87 @@
/*
* (c) Copyright 2015 Mirantis Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
(function () {
'use strict';
angular
.module('horizon.dashboard.developer.profiler')
.directive('traceTable', traceTable)
.directive('nodeData', nodeData);
nodeData.$inject = [
'$compile',
'$templateCache',
'horizon.dashboard.developer.profiler.basePath'];
function nodeData($compile, $templateCache, basePath) {
return {
restrict: 'A',
scope: {
hideChildren: '=',
rootData: '=',
data: '=nodeData',
visible: '='
},
require: '^traceTable',
link: function(scope, element, attrs, sharedCtrl) {
var destroyWatcher = scope.$watch('rootData', function(newValue) {
if (angular.isDefined(newValue)) {
var template = $templateCache.get(basePath + 'profiler.tree-node.html');
scope.ctrl = sharedCtrl;
element.replaceWith($compile(template)(scope));
}
});
scope.$on('$destroy', function() {
destroyWatcher();
});
}
}
}
traceTable.$inject = [
'horizon.framework.util.http.service',
'horizon.dashboard.developer.basePath'];
function traceTable($http, basePath) {
return {
restrict: 'A',
templateUrl: basePath + 'profiler/profiler.trace-table.html',
scope: {
trace: '=traceTable'
},
replace: true,
controller: 'sharedProfilerController',
link: function(scope) {
var destroyWatcher = scope.$watch('trace', function(trace) {
if (trace) {
var traceId = trace.id;
scope.$on('hzTable:rowExpanded', function(e, traceItem) {
if (traceId === traceItem.id && !scope.data) {
$http.get('/api/profiler/traces/' + traceId).then(function(response) {
scope.data = response.data;
});
}
});
}
});
scope.$on('$destroy', function() {
destroyWatcher();
});
}
}
}
})();

View File

@ -0,0 +1,40 @@
/*
* (c) Copyright 2016 Mirantis Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
(function () {
'use strict';
/**
* @ngdoc module
* @ngname horizon.dashboard.developer.profiler
* @description
* Dashboard module for the profiler panel.
*/
angular
.module('horizon.dashboard.developer.profiler', ['ui.bootstrap'])
.config(config);
config.$inject = [
'$provide',
'$windowProvider'
];
function config($provide, $windowProvider) {
var path = $windowProvider.$get().STATIC_URL + 'dashboard/developer/profiler/';
$provide.constant('horizon.dashboard.developer.profiler.basePath', path);
}
})();

View File

@ -0,0 +1,5 @@
.profiler {
.progress-bar-transparent {
background-color: rgba(255, 255, 255, 0.0);
}
}

View File

@ -0,0 +1,20 @@
<table class="trace table table-hover">
<thead>
<tr class="bold text-left">
<td class="level">Levels</td>
<td>Duration</td>
<td>Type</td>
<td>Project</td>
<td>Service</td>
<td>Host</td>
<td class="details">Details</td>
</tr>
</thead>
<tbody>
<tr node-data="data" root-data="data" visible="true">
<td colspan="7" align="center">
<i class="fa fa-spin fa-refresh fa-3x"></i>
</td>
</tr>
</tbody>
</table>

View File

@ -0,0 +1,28 @@
<tr ng-if="data" ng-show="visible">
<td class="level" ng-style="{'padding-left': data.level * 5 + 'px'}">
<button type="button" class="btn btn-default btn-xs"
ng-disabled="data.is_leaf" ng-click="ctrl.toggleChildren(data)">
<span class="fa" ng-class="[ctrl.getLeafCls(data), ctrl.getBranchCls(data)]"></span>
{$ data.level || 0 $}
</button>
</td>
<td>
<div class="progress-text">
<progress>
<bar value="ctrl.getStarted(data, rootData)" type="transparent"></bar>
<bar value="ctrl.getWidth(data, rootData)" type="info"></bar>
</progress>
<span class="progress-bar-text">
{$ data.info.finished - data.info.started $} ms
</span>
</div>
</td>
<td class="{$ ctrl.isImportant(data) ? 'bold' : ''$}" >{$ data.info.name $}</td>
<td>{$ data.info.project || "n/a"$}</td>
<td>{$ data.info.service || "n/a" $}</td>
<td>{$ data.info.host || "n/a"$}</td>
<td><a href="#" ng-click="ctrl.display(data);">Details</a></td>
</tr>
<tr ng-repeat="child in data.children" node-data="child" visible="child.visible"
root-data="rootData"></tr>

View File

@ -29,6 +29,9 @@
</ul>
<ul class="nav navbar-nav navbar-right">
{% if profiler_enabled %}
{% include "developer/profiler/_mode_picker.html" %}
{% endif %}
{% include "header/_user_menu.html" %}
{% include "header/_region_selection.html" %}
</ul>

View File

@ -119,6 +119,8 @@ settings_utils.update_dashboards(
INSTALLED_APPS,
)
OPENSTACK_PROFILER = {'enabled': False}
settings_utils.find_static_files(HORIZON_CONFIG, AVAILABLE_THEMES,
THEME_COLLECTION_DIR, ROOT_PATH)

View File

@ -29,6 +29,9 @@
</ul>
<ul class="nav navbar-nav navbar-right">
{% if profiler_enabled %}
{% include "developer/profiler/_mode_picker.html" %}
{% endif %}
{% include "header/_user_menu.html" %}
{% include "header/_region_selection.html" %}
</ul>

View File

@ -0,0 +1,22 @@
---
features:
- A new Profiler panel in the Developer dashboard is
introduced. It integrates
`osprofiler library <http://docs.openstack.org/developer/osprofiler/>`_
into horizon, thus implementing
`blueprint openstack-profiler-at-developer-dashboard <https://blueprints.launchpad.net/horizon/+spec/openstack-profiler-at-developer-dashboard>`_.
Initially profiler is disabled. To enable it the value
``OPENSTACK_PROFILER['enabled']`` has to be ``True``.
This in turn can be achieved by copying files
_9030_profiler_settings.py.example and _9030_profiler.py to
openstack_dashboard/local/local_settings.d/_9030_profiler_settings.py
and openstack_dashboard/local/enabled/_9030_profiler.py respectively.
Also, by default it expects MongoDB cluster
to be present on the same host where Keystone is located
(say, in a Devstack VM). But it also can be configured
with params with ``OPENSTACK_PROFILER['notifier_connection_string]'``
and ``OPENSTACK_PROFILER['receiver_connection_string']`` values.
MongoDB should be installed
`manually <https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/#install-mongodb-community-edition>`_
and allowed to receive requests on 0.0.0.0 interface.