openstack/storlets
Code Issues Proposed changes

Adding invocation magics to the IPython extension

Browse Source 
This commit adds:
1. A dedicated documentation for the IPython extension
2. A %get command for storlet invocation on get
3. A %put command for storlet invocation on put
4. A %copy command for storlet invocation on copy

The syntax is documented in /doc/source/ipython_integration.rst

Bonus: adding __init__.py to the extension's unit test

Change-Id: Idf09aa03222586a39dfec7d132554d14f908e200

Refactor IPython extension and the tests

Mainly:

- Implements Response class instead of response dict
  (need to update the docs)

- Fix FakeConnection class from the redandunt code perspective

Need to more generarize around _call_xx in the tests...

Change-Id: Ieb46c7a696e8a4c2fe3a9cc0858ab5fef12678c6

Cleanup test_ipython code doc update and nb for test

- Abstract BaseTestIpythonExtension class for testing tools
-- And then, parse the child test classes for the methods to assert
- Remove unnecessary args and mock patches
- Avoid redandunt code duplication

Update the docs with the Response

Adding a notebook for testing the ipython extension
plus automated testing of that notebook

Change-Id: I0842fae5f20268cdd8ccf284eecab77498074e83
This commit is contained in:
Eran Rom 2017-01-02 22:53:10 +02:00 committed by Kota Tsuyuzaki
parent ef80317dc2
commit f21244c0b2
11 changed files with 1118 additions and 122 deletions

3
.gitignore vendored

@ -47,6 +47,9 @@ StorletSamples/java/*/bin
# scripts build
scripts/restart_docker_container
# functional tests
tests/functional/.ipynb_checkpoints/
# Unit test / coverage reports
.coverage
.tox

1
doc/source/index.rst

@ -51,6 +51,7 @@ Overview and Concepts
storlets_terminology
storlet_engine_overview
api/overview_api
ipython_integration
Storlet Engine Developers
=========================

164
doc/source/ipython_integration.rst Normal file

@ -0,0 +1,164 @@
IPython Notebook Integration
============================
IPython/Jupyter provides a browser based interactive shell that supports data visualization. The storlets integration with IPython allows an easy deployment and invocation of storlets via an IPython notebook. In the below sections we describe how to setup IPython notebook to work with storlets, how to deploy a python storlet and how to invoke a storlet.
Set up IPython to work with storlets
------------------------------------
Setting up an IPython notebook to work with storlets involves:
#. Providing the authentication information of a storlet enabled Swift account.
This is done by setting environment variables similar to those used by swift
client. The exact variables that need to be set are dependent on the auth middleware
used and the auth protocol version. For more details please refer to:
`python-swiftclient docs
<http://docs.openstack.org/developer/python-swiftclient/cli.html#authentication>`_.
#. Load the storlets IPython extension.
The below shows environment variables definitions that comply with the
default storlets development environment installation (`s2aio <http://storlets.readthedocs.io/en/latest/s2aio.html>`__).
::
import os
os.environ['OS_AUTH_VERSION'] = '3'
os.environ['OS_AUTH_URL'] = 'http://127.0.0.1:5000/v3'
os.environ['OS_USERNAME'] = 'tester'
os.environ['OS_PASSWORD'] = 'testing'
os.environ['OS_USER_DOMAIN_NAME'] = 'default'
os.environ['OS_PROJECT_DOMAIN_NAME'] = 'default'
os.environ['OS_PROJECT_NAME'] = 'test'
To load the storlets IPython extension simply enter and execute the below:
::
%load_ext storlets.tools.extensions.ipython
Deploy a Python storlet
-----------------------
General background on storlets deployment is found `here <http://storlets.readthedocs.io/en/latest/writing_and_deploying_storlets.html#storlet-deployment-guidelines>`__.
In a new notebook cell, enter the '%%storletapp' directive
followed by the storlet name. Followng that type the storlet code.
Below is an example of a simple 'identitiy' storlet.
Executing the cell will deploy the storlet into Swift.
::
%%storletapp test.TestStorlet
class TestStorlet(object):
def __init__(self, logger):
self.logger = logger
def __call__(self, in_files, out_files, params):
"""
The function called for storlet invocation
:param in_files: a list of StorletInputFile
:param out_files: a list of StorletOutputFile
:param params: a dict of request parameters
"""
self.logger.debug('Returning metadata')
metadata = in_files[0].get_metadata()
for key in params.keys():
metadata[key] = params[key]
out_files[0].set_metadata(metadata)
self.logger.debug('Start to return object data')
content = ''
while True:
buf = in_files[0].read(16)
if not buf:
break
content += buf
self.logger.debug('Recieved %d bytes' % len(content))
self.logger.debug('Writing back %d bytes' % len(content))
out_files[0].write(content)
self.logger.debug('Complete')
in_files[0].close()
out_files[0].close()
.. note:: To run the storlet on an actual data set, one can enter the following at
the top of the cell
::
%%storletapp test.TestStorlet --with-invoke --input path:/<container>/<object> --print-result
N.B. Useful commands such as 'dry-run' is under development. And more
details for options are in the next section.
Invoke a storlet
----------------
General information on storlet invocation can be found `here <http://storlets.readthedocs.io/en/latest/api/overview_api.html#storlets-invocation>`__.
Here is how an invocation works:
#. Define an optional dictionay variable params that would hold the invocation parameters:
::
myparams = {'color' : 'red'}
#. To invoke test.TestStorlet on a get just type the following:
::
%get --storlet test.py --input path:/<container>/<object> -i myparams -o myresult
The invocation will execute test.py over the specified swift object with parameters read from myparams.
The result is placed in myresults.
The '-i' argument is optional, however, if specified the supplied value must be a name of a defined dictionary variable.
myresults is an instance of storlets.tools.extensions.ipython.Response. This class has the following members:
#. status - An integer holding the Http response status
#. headers - A dictionary holding the storlet invocation response headers
#. iter_content - An iterator over the response body
#. content - The content of the response body
#. To invoke test.TestStorlet on a put just type the following:
::
%put --storlet test.py --input <full path to local file> --output path:/<container>/<object> -i myparams -o myresult
The invocation will execute test.py over the uploaded file specified with the --input option which must be a full local path.
test.py is invoked with parameters read from myparams.
The result is placed in myresults.
The '-i' argument is optional, however, if specified the supplied value must be a name of a defined variable.
myresults is a dictionary with the following keys:
#. status - An integer holding the Http response status
#. headers - A dictionary holding the storlet invocation response headers
#. To invoke test.TestStorlet on a copy just type the following:
::
%copy --storlet test.py --input path:/<container>/<object> --output path:/<container>/<object> -i myparams -o myresult
The invocation will execute test.py over the input object specified with the --input option.
The execution result will be saved in the output object specified with the --output option.
test.py is invoked with parameters read from myparams.
The result is placed in myresults.
The '-i' argument is optional, however, if specified the supplied value must be a name of a defined variable.
myresults is a dictionary with the following keys:
#. status - An integer holding the Http response status
#. headers - A dictionary holding the storlet invocation response headers
Extension docs
^^^^^^^^^^^^^^
.. automodule:: storlets.tools.extensions.ipython
:members:
:show-inheritance:

93
doc/source/writing_and_deploying_python_storlets.rst

@ -81,96 +81,3 @@ Deploying a Python Dependency
-----------------------------
#. Currently, there is no limitation as to what is being uploaded as a dependency.
Writing Storlet App with IPython/Jupyter Notebook
-------------------------------------------------
Storlets supports IPython/Jupyter Notebookd extension to upload your own
storlet apps with the following steps:
.. note::
To upload a storlet app to Swift one needs to provide the authentication
information of the Storlet enabled Swift account. This is done by setting
environment variables similar to those used by swift client. The exact
variables that need to be set are dependent on the auth middleware used
and the auth protocol version. For more details please refer to
`python-swiftclient docs
<http://docs.openstack.org/developer/python-swiftclient/cli.html#authentication>`_.
In case you are working with an s2aio,sh installation just add a new cell
with the following:
::
import os
os.environ['OS_AUTH_VERSION'] = '3'
os.environ['OS_AUTH_URL'] = 'http://127.0.0.1:5000/v3'
os.environ['OS_USERNAME'] = 'tester'
os.environ['OS_PASSWORD'] = 'testing'
os.environ['OS_USER_DOMAIN_NAME'] = 'default'
os.environ['OS_PROJECT_DOMAIN_NAME'] = 'default'
os.environ['OS_PROJECT_NAME'] = 'test'
1. Enables storlets extension on your app
::
%reload_ext storlets.tools.extensions.ipython
2. Add another cell for your app and add storletapp command to the top of the
cell
::
%%storletapp test.TestStorlet
class TestStorlet(object):
def __init__(self, logger):
self.logger = logger
def __call__(self, in_files, out_files, params):
"""
The function called for storlet invocation
:param in_files: a list of StorletInputFile
:param out_files: a list of StorletOutputFile
:param params: a dict of request parameters
"""
self.logger.debug('Returning metadata')
metadata = in_files[0].get_metadata()
metadata['test'] = 'simple'
out_files[0].set_metadata(metadata)
self.logger.debug('Start to return object data')
while True:
buf = in_files[0].read(16)
if not buf:
break
self.logger.debug('Recieved %d bytes' % len(buf))
self.logger.debug('Writing back %d bytes' % len(buf))
out_files[0].write(buf)
self.logger.debug('Complete')
in_files[0].close()
out_files[0].close()
3. If you want to run the app with actual data set, please specify some options
like:
::
%%storletapp test.TestStorlet --with-invoke --input path:/<container>/<object> --print-result
N.B. the useful commands like 'dry-run', etc... is under development. And more
details for options are in the next section.
Extension docs
^^^^^^^^^^^^^^
.. automodule:: storlets.tools.extensions.ipython
:members:
:show-inheritance:

5
storlets/swift_middleware/handlers/proxy.py

@ -238,7 +238,10 @@ class StorletProxyHandler(StorletBaseHandler):
resp = storlet_req.get_response(self.app)
if not resp.is_success:
raise HTTPUnauthorized('Failed to verify access to the storlet',
raise HTTPUnauthorized('Failed to verify access to the storlet. '
'Either the storlet does not exist or '
'you are not authorized to run the '
'storlet.',
request=self.request)
params = self._parse_storlet_params(resp.headers)

285
storlets/tools/extensions/ipython.py

@ -23,6 +23,7 @@ authentication and storage target host. (for now)
from __future__ import print_function
import os
import string
from swiftclient.client import Connection
from IPython.core import magic_arguments
@ -30,10 +31,36 @@ from IPython.core import magic_arguments
# errors import as references.
# from IPython.core.alias import AliasError, Alias
from IPython.core.error import UsageError
from IPython.core.magic import Magics, magics_class, cell_magic
from IPython.core.magic import Magics, magics_class, cell_magic, line_magic
from IPython.utils.py3compat import unicode_type
class Response(object):
"""
Response object to return the object to ipython cell
:param status: int for status code
:param headers: a dict for repsonse headers
:param body_iter: an iterator object which takes the body content from
"""
def __init__(self, status, headers, body_iter=None):
self.status = status
self.headers = headers
self._body_iter = body_iter or iter([])
def __iter__(self):
print('hoge')
return self._body_iter
def iter_content(self):
# TODO(kota_): supports chunk_size like requests.Response
return self._body_iter
@property
def content(self):
return ''.join([chunk for chunk in self._body_iter])
def get_swift_connection():
# find api version
for k in ('ST_AUTH_VERSION', 'OS_AUTH_VERSION', 'OS_IDENTITY_API_VERSION'):
@ -52,7 +79,7 @@ def get_swift_connection():
auth_url = os.environ['OS_AUTH_URL']
auth_user = os.environ['OS_USERNAME']
auth_password = os.environ['OS_PASSWORD']
project_name = os.environ['OS_PROJECT_NAME'],
project_name = os.environ['OS_PROJECT_NAME']
except KeyError:
raise UsageError(
"You need to set OS_AUTH_URL, OS_USERNAME, OS_PASSWORD and "
@ -88,6 +115,43 @@ class StorletMagics(Magics):
"""Magics to interact with OpenStack Storlets
"""
def _parse_input_path(self, path_str):
"""
Parse formatted to path to swift container and object names
:param path_str: path string starts with "path:" prefix
:return (container, obj): Both container and obj are formatted
as string
:raise UsageError: if the path_str is not formatted as expected
"""
if not path_str.startswith('path:'):
raise UsageError(
'swift object path must have the format: '
'"path:/<container>/<object>"')
try:
src_container_obj = path_str[len('path:'):]
src_container, src_obj = src_container_obj.strip(
'/').split('/', 1)
return src_container, src_obj
except ValueError:
raise UsageError(
'swift object path must have the format: '
'"path:/<container>/<object>"')
def _generate_params_headers(self, param_dict):
"""
Parse parameter args dict to swift headers
:param param_dict: a dict of input parameters
:return headers: a dict for swift headers
"""
headers = {}
for i, (key, value) in enumerate(param_dict.items()):
headers['X-Storlet-Parameter-%d' % i] =\
'%s:%s' % (key, value)
return headers
@magic_arguments.magic_arguments()
@magic_arguments.argument(
'container_obj', type=unicode_type,
@ -157,18 +221,8 @@ class StorletMagics(Magics):
if not args.input:
raise UsageError(
'--with-invoke option requires --input to run the app')
if not args.input.startswith('path:'):
raise UsageError(
'--input option for --with-invoke must be path format '
'"path:/<container>/<object>"')
try:
src_container_obj = args.input[len('path:'):]
src_container, src_obj = src_container_obj.strip(
'/').split('/', 1)
except ValueError:
raise UsageError(
'--input option for --with-invoke must be path format '
'"path:/<container>/<object>"')
src_container, src_obj = self._parse_input_path(args.input)
headers = {'X-Run-Storlet': '%s' % storlet_obj}
@ -186,6 +240,209 @@ class StorletMagics(Magics):
for x in resp_content_iter:
pass
@magic_arguments.magic_arguments()
@magic_arguments.argument(
'--input', type=unicode_type,
help='The input object for the storlet execution'
'this option must be of the form "path:<container>/<object>"'
)
@magic_arguments.argument(
'--storlet', type=unicode_type,
help='The storlet to execute over the input'
)
@magic_arguments.argument(
'-i', type=unicode_type,
help=('A name of a variable defined in the environment '
'holding a dictionary with the storlet invocation '
'input parameters')
)
@magic_arguments.argument(
'-o', type=unicode_type,
help=('A name of an output variable to hold the invocation result '
'The output variable is a dictionary with the fields: '
'status, headers, content_iter holding the reponse status, '
'headers, and body iterator accordingly')
)
@line_magic
def get(self, line):
args = magic_arguments.parse_argstring(self.get, line)
if not args.o:
raise UsageError('-o option is mandatory for the invocation')
if not args.o[0].startswith(tuple(string.ascii_letters)):
raise UsageError('The output variable name must be a valid prefix '
'of a python variable, that is, start with a '
'letter')
if not args.storlet:
raise UsageError('--storlet option is mandatory '
'for the invocation')
if not args.input:
raise UsageError('--input option is mandatory for the invocation')
src_container, src_obj = self._parse_input_path(args.input)
headers = {'X-Run-Storlet': '%s' % args.storlet}
# pick -i option and translate the params to
# X-Storlet-Parameter-x headers
storlet_headers = self._generate_params_headers(
self.shell.user_ns[args.i] if args.i else {})
headers.update(storlet_headers)
# invoke storlet app on get
conn = get_swift_connection()
response_dict = dict()
resp_headers, resp_content_iter = conn.get_object(
src_container, src_obj,
resp_chunk_size=64 * 1024,
headers=headers,
response_dict=response_dict)
res = Response(int(response_dict['status']),
resp_headers,
resp_content_iter)
self.shell.user_ns[args.o] = res
@magic_arguments.magic_arguments()
@magic_arguments.argument(
'--input', type=unicode_type,
help='The input object for the storlet execution'
'this option must be of the form "path:<container>/<object>"'
)
@magic_arguments.argument(
'--output', type=unicode_type,
help='The output object for the storlet execution'
'this option must be of the form "path:<container>/<object>"'
)
@magic_arguments.argument(
'--storlet', type=unicode_type,
help='The storlet to execute over the input'
)
@magic_arguments.argument(
'-i', type=unicode_type,
help=('A name of a variable defined in the environment '
'holding a dictionary with the storlet invocation '
'input parameters')
)
@magic_arguments.argument(
'-o', type=unicode_type,
help=('A name of an output variable to hold the invocation result '
'The output variable is a dictionary with the fields: '
'status, headers, holding the reponse status and '
'headers accordingly')
)
@line_magic
def copy(self, line):
args = magic_arguments.parse_argstring(self.copy, line)
if not args.o:
raise UsageError('-o option is mandatory for the invocation')
if not args.o[0].startswith(tuple(string.ascii_letters)):
raise UsageError('The output variable name must be a valid prefix '
'of a python variable, that is, start with a '
'letter')
if not args.storlet:
raise UsageError('--storlet option is mandatory '
'for the invocation')
if not args.input:
raise UsageError('--input option is mandatory for the invocation')
if not args.output:
raise UsageError('--output option is mandatory for the invocation')
src_container, src_obj = self._parse_input_path(args.input)
dst_container, dst_obj = self._parse_input_path(args.output)
destination = '/%s/%s' % (dst_container, dst_obj)
headers = {'X-Run-Storlet': '%s' % args.storlet}
# pick -i option and translate the params to
# X-Storlet-Parameter-x headers
storlet_headers = self._generate_params_headers(
self.shell.user_ns[args.i] if args.i else {})
headers.update(storlet_headers)
# invoke storlet app on copy
conn = get_swift_connection()
response_dict = dict()
conn.copy_object(
src_container, src_obj,
destination=destination,
headers=headers,
response_dict=response_dict)
res = Response(int(response_dict['status']),
response_dict['headers'])
self.shell.user_ns[args.o] = res
@magic_arguments.magic_arguments()
@magic_arguments.argument(
'--input', type=unicode_type,
help='The local input object for upload'
'this option must be a full path of a local file'
)
@magic_arguments.argument(
'--output', type=unicode_type,
help='The output object of the storlet execution'
'this option must be of the form "path:<container>/<object>"'
)
@magic_arguments.argument(
'--storlet', type=unicode_type,
help='The storlet to execute over the input'
)
@magic_arguments.argument(
'-i', type=unicode_type,
help=('A name of a variable defined in the environment '
'holding a dictionary with the storlet invocation '
'input parameters')
)
@magic_arguments.argument(
'-o', type=unicode_type,
help=('A name of an output variable to hold the invocation result '
'The output variable is a dictionary with the fields: '
'status, headers, holding the reponse status and '
'headers accordingly')
)
@line_magic
def put(self, line):
args = magic_arguments.parse_argstring(self.put, line)
if not args.o:
raise UsageError('-o option is mandatory for the invocation')
if not args.o[0].startswith(tuple(string.ascii_letters)):
raise UsageError('The output variable name must be a valid prefix '
'of a python variable, that is, start with a '
'letter')
if not args.storlet:
raise UsageError('--storlet option is mandatory '
'for the invocation')
if not args.input:
raise UsageError('--input option is mandatory for the invocation')
if not args.input.startswith('/'):
raise UsageError('--input argument must be a full path')
if not args.output:
raise UsageError('--output option is mandatory for the invocation')
dst_container, dst_obj = self._parse_input_path(args.output)
headers = {'X-Run-Storlet': '%s' % args.storlet}
# pick -i option and translate the params to
# X-Storlet-Parameter-x headers
storlet_headers = self._generate_params_headers(
self.shell.user_ns[args.i] if args.i else {})
headers.update(storlet_headers)
# invoke storlet app on copy
conn = get_swift_connection()
response_dict = dict()
with open(args.input, 'r') as content:
conn.put_object(
dst_container, dst_obj,
content,
headers=headers,
response_dict=response_dict)
res = Response(int(response_dict['status']),
response_dict['headers'])
self.shell.user_ns[args.o] = res
def load_ipython_extension(ipython):
ipython.register_magics(StorletMagics)

2
test-requirements.txt

@ -16,6 +16,8 @@ testtools>=0.9.36,!=1.2.0
python-swiftclient
python-keystoneclient
ipython<6.0
jupyter
nbformat
ansible
ansible-lint

0
tests/functional/ipython/__init__.py Normal file

128
tests/functional/ipython/test_jupyter_execution.py Normal file

@ -0,0 +1,128 @@
# Copyright (c) 2010-2016 OpenStack Foundation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import unittest
import tempfile
import subprocess
import nbformat
import six
COULD_BE_CHANGED = ['x-storlet-generated-from-account',
'x-trans-id',
'x-openstack-request-id',
'x-storlet-generated-from-last-modified',
'last-modified',
'x-timestamp',
'date', ]
class TestJupyterExcecution(unittest.TestCase):
def _run_notebook(self, path):
"""Execute a notebook via nbconvert and collect output.
:returns (parsed nb object, execution errors)
"""
with tempfile.NamedTemporaryFile(suffix=".ipynb") as fout:
args = ["jupyter", "nbconvert", "--to", "notebook", "--execute",
"--ExecutePreprocessor.timeout=60",
"--output", fout.name, path]
try:
subprocess.check_output(args, stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
# Note that CalledProcessError will have stdout/stderr in py3
# instead of output attribute
self.fail('jupyter nbconvert fails with:\n'
'STDOUT: %s\n' % (e.output))
fout.seek(0)
nb = nbformat.read(fout, nbformat.current_nbformat)
# gather all error messages in all cells in the notebook
errors = [output for cell in nb.cells if "outputs" in cell
for output in cell["outputs"]
if output.output_type == "error"]
return nb, errors
def _clear_text_list(self, node):
"""
convert a notebook cell to text list like ["a", "b"]
N.B. each text will be striped
"""
texts = list()
if 'text' in node:
for line in node['text'].split('\n'):
if line:
texts.append(line.strip())
return texts
return None
def _flatten_output_text(self, notebook):
"""
This helper method make the notebook output cells flatten to a single
direction list.
"""
output_text_list = []
for cell in notebook.cells:
for output in cell.get("outputs", []):
output_text_list.extend(self._clear_text_list(output))
return output_text_list
def test_notebook(self):
test_path = os.path.abspath(__file__)
test_dir = os.path.dirname(test_path)
original_notebook = os.path.join(test_dir, 'test_notebook.ipynb')
with open(original_notebook) as f:
original_nb = nbformat.read(f, nbformat.current_nbformat)
expected_output = self._flatten_output_text(original_nb)
got_nb, errors = self._run_notebook(original_notebook)
self.assertFalse(errors)
got = self._flatten_output_text(got_nb)
self._assert_output(expected_output, got)
def _assert_output(self, expected_output, got):
for expected_line, got_line in zip(expected_output, got):
try:
expected_line = eval(expected_line)
got_line = eval(got_line)
except (NameError, SyntaxError, AttributeError):
# sanity, both line should be string type
self.assertIsInstance(expected_line, six.string_types)
self.assertIsInstance(got_line, six.string_types)
# this is for normal text line (NOT json dict)
self.assertEqual(expected_line, got_line)
else:
if isinstance(expected_line, dict) and \
isinstance(got_line, dict):
expected_and_got = zip(
sorted(expected_line.items()),
sorted(got_line.items()))
for (expected_key, expected_value), (got_key, got_value) in \
expected_and_got:
self.assertEqual(expected_key, got_key)
if expected_key in COULD_BE_CHANGED:
# TODO(kota_): make more validation for each format
continue
else:
self.assertEqual(expected_value, got_value)
else:
self.assertEqual(expected_line, got_line)
if __name__ == '__main__':
unittest.main()

221
tests/functional/ipython/test_notebook.ipynb Normal file

@ -0,0 +1,221 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": true,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"%reload_ext storlets.tools.extensions.ipython\n",
"import os\n",
"os.environ['OS_AUTH_VERSION'] = '3'\n",
"os.environ['OS_AUTH_URL'] = 'http://127.0.0.1/identity/v3'\n",
"os.environ['OS_USERNAME'] = 'tester'\n",
"os.environ['OS_PASSWORD'] = 'testing'\n",
"os.environ['OS_USER_DOMAIN_NAME'] = 'default'\n",
"os.environ['OS_PROJECT_DOMAIN_NAME'] = 'default'\n",
"os.environ['OS_PROJECT_NAME'] = 'test'"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Upload storlets succeeded /storlet/test.py\n",
"Example command `swift download <container> <object> -H X-Run-Storlet:test.py`\n"
]
}
],
"source": [
"%%storletapp test.TestStorlet\n",
"\n",
"class TestStorlet(object):\n",
" def __init__(self, logger):\n",
" self.logger = logger\n",
"\n",
" def __call__(self, in_files, out_files, params):\n",
" \"\"\"\n",
" The function called for storlet invocation\n",
" :param in_files: a list of StorletInputFile\n",
" :param out_files: a list of StorletOutputFile\n",
" :param params: a dict of request parameters\n",
" \"\"\"\n",
" self.logger.debug('Returning metadata')\n",
" metadata = in_files[0].get_metadata()\n",
" for key in params.keys():\n",
" metadata[key] = params[key]\n",
" out_files[0].set_metadata(metadata)\n",
"\n",
" self.logger.debug('Start to return object data')\n",
" content = ''\n",
" while True:\n",
" buf = in_files[0].read(16)\n",
" if not buf:\n",
" break\n",
" content += buf\n",
" self.logger.debug('Recieved %d bytes' % len(content))\n",
" self.logger.debug('Writing back %d bytes' % len(content))\n",
" out_files[0].write(content)\n",
" self.logger.debug('Complete')\n",
" in_files[0].close()\n",
" out_files[0].close()"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{u'x-object-meta-storlet-language': u'python', u'x-trans-id': u'tx39de631f32cf42d58f407-0058f89455', u'x-object-meta-storlet-main': u'test.TestStorlet', u'transfer-encoding': u'chunked', u'x-object-meta-storlet-interface-version': u'1.0', u'x-object-meta-storlet-object-metadata': u'no', u'x-object-meta-storlet_execution_path': u'/home/swift/test.TestStorlet', u'last-modified': u'Thu, 20 Apr 2017 10:58:29 GMT', u'etag': u'f4b96093d75348b4c55c2403f38f0700', u'x-timestamp': u'1492685908.66304', u'x-object-meta-color': u'red', u'date': u'Thu, 20 Apr 2017 10:58:29 GMT', u'x-openstack-request-id': u'tx39de631f32cf42d58f407-0058f89455', u'content-type': u'application/octet-stream', u'accept-ranges': u'bytes'}\n",
"\n",
"class TestStorlet(object):\n",
" def __init__(self, logger):\n",
" self.logger = logger\n",
"\n",
" def __call__(self, in_files, out_files, params):\n",
" \"\"\"\n",
" The function called for storlet invocation\n",
" :param in_files: a list of StorletInputFile\n",
" :param out_files: a list of StorletOutputFile\n",
" :param params: a dict of request parameters\n",
" \"\"\"\n",
" self.logger.debug('Returning metadata')\n",
" metadata = in_files[0].get_metadata()\n",
" for key in params.keys():\n",
" metadata[key] = params[key]\n",
" out_files[0].set_metadata(metadata)\n",
"\n",
" self.logger.debug('Start to return object data')\n",
" content = ''\n",
" while True:\n",
" buf = in_files[0].read(16)\n",
" if not buf:\n",
" break\n",
" content += buf\n",
" self.logger.debug('Recieved %d bytes' % len(content))\n",
" self.logger.debug('Writing back %d bytes' % len(content))\n",
" out_files[0].write(content)\n",
" self.logger.debug('Complete')\n",
" in_files[0].close()\n",
" out_files[0].close()\n"
]
}
],
"source": [
"myparams = {'color' : 'red'}\n",
"%get --storlet test.py --input path:/storlet/test.py -i myparams -o myresult\n",
"print myresult.headers\n",
"print myresult.content"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{u'content-length': u'0', u'x-storlet-generated-from-last-modified': u'Thu, 20 Apr 2017 10:58:29 GMT', u'x-storlet-generated-from-account': u'AUTH_6dbd182dfa9f4ad6ace88992683ee483', u'last-modified': u'Thu, 20 Apr 2017 10:58:32 GMT', u'etag': u'f4b96093d75348b4c55c2403f38f0700', u'x-trans-id': u'tx8cc66cd8cda643508bee7-0058f89457', u'date': u'Thu, 20 Apr 2017 10:58:31 GMT', u'content-type': u'text/html; charset=UTF-8', u'x-openstack-request-id': u'tx8cc66cd8cda643508bee7-0058f89457', u'x-storlet-generated-from': u'storlet/test.py'}\n",
"201\n"
]
}
],
"source": [
"%copy --storlet test.py --input path:/storlet/test.py --output path:/log/test.py -i myparams -o myresult\n",
"print myresult.headers\n",
"print myresult.status"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{u'content-length': u'0', u'last-modified': u'Thu, 20 Apr 2017 10:58:34 GMT', u'etag': u'faccc93089c22cafce9b64e7cc9f2047', u'x-trans-id': u'tx9577116d45ee4e43bc3fd-0058f89458', u'date': u'Thu, 20 Apr 2017 10:58:33 GMT', u'content-type': u'text/html; charset=UTF-8', u'x-openstack-request-id': u'tx9577116d45ee4e43bc3fd-0058f89458'}\n",
"201\n"
]
}
],
"source": [
"import os\n",
"try:\n",
" os.mkdir('/tmp/tmpZl6teg')\n",
"except:\n",
" pass\n",
"with open('/tmp/tmpZl6teg/storlet_invoke.log', 'w') as f:\n",
" for x in range(10):\n",
" f.write('INFO: sapmle log line')\n",
"%put --storlet test.py --input /tmp/tmpZl6teg/storlet_invoke.log --output path:/log/onvoke.log -i myparams -o myresult\n",
"print myresult.headers\n",
"print myresult.status"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"deletable": true,
"editable": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.12"
}
},
"nbformat": 4,
"nbformat_minor": 2
}

338
tests/unit/tools/extensions/test_ipython.py

@ -22,18 +22,52 @@ import os
import itertools
class FakeConnection(mock.MagicMock):
class FakeConnection(object):
def __init__(self, fake_status=200, fake_headers=None, fake_iter=None):
self._fake_status = fake_status
self._fake_headers = fake_headers or {}
self._fake_iter = fake_iter or iter([])
def _return_fake_response(self, **kwargs):
if 'response_dict' in kwargs:
kwargs['response_dict']['status'] = self._fake_status
kwargs['response_dict']['headers'] = self._fake_headers
kwargs['response_dict']['content_iter'] = self._fake_iter
if 'resp_chunk_size' in kwargs:
resp_body = self._fake_iter
else:
resp_body = ''.join([chunk for chunk in self._fake_iter])
return (self._fake_headers, resp_body)
# Those 3 methods are just for entry point difference from the caller
# but all methods returns same response format with updateing response_dict
def get_object(self, *args, **kwargs):
return (mock.MagicMock(), mock.MagicMock())
return self._return_fake_response(**kwargs)
def copy_object(self, *args, **kwargs):
return self._return_fake_response(**kwargs)
def put_object(self, *args, **kwargs):
return self._return_fake_response(**kwargs)
class TestStorletMagics(unittest.TestCase):
class MockShell(object):
def __init__(self):
self.user_ns = {}
def register(self, var_name, value):
self.user_ns[var_name] = value
class BaseTestIpythonExtension(object):
def setUp(self):
self.fake_connection = FakeConnection()
self.magics = StorletMagics()
# set auth info for keystone
self.os_original_env = os.environ.copy()
self._set_auth_environ()
self.magics.shell = MockShell()
def tearDown(self):
os.environ = self.os_original_env.copy()
@ -48,12 +82,26 @@ class TestStorletMagics(unittest.TestCase):
os.environ['OS_PROJECT_DOMAIN_NAME'] = 'default'
os.environ['OS_PROJECT_NAME'] = 'test'
@mock.patch('storlets.tools.extensions.ipython.Connection')
def _call_cell(self, func, line, cell, fake_conn):
fake_conn.return_value = self.fake_connection
# cell magic
func(line, cell)
@mock.patch('storlets.tools.extensions.ipython.Connection')
def _call_line(self, func, line, fake_conn):
fake_conn.return_value = self.fake_connection
# line magic
func(line)
class TestStorletMagicStorletApp(BaseTestIpythonExtension, unittest.TestCase):
def setUp(self):
super(TestStorletMagicStorletApp, self).setUp()
self.fake_connection = FakeConnection()
def _call_storletapp(self, line, cell):
# wrap up the get_swift_connection always return mock connection
with mock.patch(
'storlets.tools.extensions.ipython.Connection') as fake_conn:
fake_conn.return_value = self.fake_connection
self.magics.storletapp(line, cell)
self._call_cell(self.magics.storletapp, line, cell)
def test_storlet_magics(self):
line = 'test.TestStorlet'
@ -81,8 +129,7 @@ class TestStorletMagics(unittest.TestCase):
'--with-invoke option requires --input to run the app',
cm.exception.message)
def test_storlet_magics_with_invoke_invalid_input_fail(self):
cell = ''
def test_storlet_magics_invalid_input_fail(self):
invalid_input_patterns = (
'invalid', # no "path:" prefix
'path://', # no container object in the slash
@ -91,11 +138,10 @@ class TestStorletMagics(unittest.TestCase):
)
for invalid_input in invalid_input_patterns:
line = 'test.TestStorlet --with-invoke --input %s' % invalid_input
with self.assertRaises(UsageError) as cm:
self._call_storletapp(line, cell)
self.magics._parse_input_path(invalid_input)
self.assertEqual(
'--input option for --with-invoke must be path format '
'swift object path must have the format: '
'"path:/<container>/<object>"',
cm.exception.message)
@ -163,6 +209,10 @@ class TestStorletMagics(unittest.TestCase):
# v1 doesn't require OS_AUTH_VERSION
del os.environ['OS_AUTH_VERSION']
try:
del os.environ['OS_IDENTITY_API_VERSION']
except Exception:
pass
def _set_v1_auth():
os.environ['ST_AUTH'] = 'http://localhost/v1/auth'
@ -186,5 +236,265 @@ class TestStorletMagics(unittest.TestCase):
e.exception.message)
class TestStorletMagicGet(BaseTestIpythonExtension, unittest.TestCase):
def setUp(self):
super(TestStorletMagicGet, self).setUp()
self.fake_connection = FakeConnection()
def _call_get(self, line):
self._call_line(self.magics.get, line)
def test_get_invalid_args(self):
scenarios = [
{
'line': '--input path:/c/o --storlet a.b',
'exception': UsageError,
'msg': '-o option is mandatory for the invocation'
}, {
'line': '--input path:/c/o -o a1234',
'exception': UsageError,
'msg': '--storlet option is mandatory for the invocation'
}, {
'line': '--storlet a.b -o a1234',
'exception': UsageError,
'msg': '--input option is mandatory for the invocation'
}, {
'line': '--input path/c/o --storlet a.b -o a1234',
'exception': UsageError,
'msg': ('swift object path must have the format: '
'"path:/<container>/<object>"')
}, {
'line': '--input path:/c/ --storlet a.b -o a1234',
'exception': UsageError,
'msg': ('swift object path must have the format: '
'"path:/<container>/<object>"')
}, {
'line': '--input path:/c --storlet a.b -o a1234',
'exception': UsageError,
'msg': ('swift object path must have the format: '
'"path:/<container>/<object>"')
}, {
'line': '--input path:/c/o --storlet a.b -o 1234',
'exception': UsageError,
'msg': ('The output variable name must be a valid prefix '
'of a python variable, that is, start with a '
'letter')
}]
for scenario in scenarios:
with self.assertRaises(UsageError) as e:
self._call_get(scenario['line'])
self.assertEqual(scenario['msg'], e.exception.message)
def _test_get(self, line, outvar_name):
self._call_get(line)
self.assertTrue(outvar_name in self.magics.shell.user_ns)
resp = self.magics.shell.user_ns[outvar_name]
self.assertEqual({}, resp.headers)
self.assertEqual(200, resp.status)
self.assertEqual('', ''.join([chunk for chunk in iter(resp)]))
self.assertEqual('', resp.content)
def test_get(self):
outvar_name = 'a1234'
line = '--input path:/c/o --storlet a.b -o %s' % outvar_name
self._test_get(line, outvar_name)
def test_get_with_input(self):
params_name = 'params'
outvar_name = 'a1234'
line = '--input path:/c/o --storlet a.b -o %s -i %s' % (outvar_name,
params_name)
# register the variable to user_ns
self.magics.shell.register(params_name, {'a': 'b'})
self._test_get(line, outvar_name)
def test_get_with_input_error(self):
params_name = 'params'
outvar_name = 'a1234'
line = '--input path:/c/o --storlet a.b -o %s -i %s' % (outvar_name,
params_name)
with self.assertRaises(KeyError):
self._test_get(line, outvar_name)
class TestStorletMagicCopy(BaseTestIpythonExtension, unittest.TestCase):
def setUp(self):
super(TestStorletMagicCopy, self).setUp()
self.fake_connection = FakeConnection()
def _call_copy(self, line):
self._call_line(self.magics.copy, line)
def test_copy_invalid_args(self):
scenarios = [
{
'line': '--input path:/c/o --storlet a.b --output path:/c/o',
'exception': UsageError,
'msg': '-o option is mandatory for the invocation'
}, {
'line': ('--input path:/c/o --storlet a.b -o 1234 '
'--output path:/c/o'),
'exception': UsageError,
'msg': ('The output variable name must be a valid prefix '
'of a python variable, that is, start with a '
'letter')
}, {
'line': '--input path:/c/o -o a1234 --output path:/c/o',
'exception': UsageError,
'msg': '--storlet option is mandatory for the invocation'
}, {
'line': '--storlet a.b -o a1234 --output path:/c/o',
'exception': UsageError,
'msg': '--input option is mandatory for the invocation'
}, {
'line': ('--input path/c/o --storlet a.b -o a1234 '
'--output path:/c/o'),
'exception': UsageError,
'msg': ('swift object path must have the format: '
'"path:/<container>/<object>"')
}, {
'line': ('--input path:/c/ --storlet a.b -o a1234 '
'--output path:/c/o'),
'exception': UsageError,
'msg': ('swift object path must have the format: '
'"path:/<container>/<object>"')
}, {
'line': ('--input path:/c --storlet a.b -o a1234 '
'--output path:/c/o'),
'exception': UsageError,
'msg': ('swift object path must have the format: '
'"path:/<container>/<object>"')
}, {
'line': '--input path:/c --storlet a.b -o a1234 ',
'exception': UsageError,
'msg': ('--output option is mandatory for the invocation')
}, {
'line': ('--input path:/c --storlet a.b -o a1234 '
'--output path/c/o'),
'exception': UsageError,
'msg': ('swift object path must have the format: '
'"path:/<container>/<object>"')
}]
for scenario in scenarios:
with self.assertRaises(UsageError) as e:
self._call_copy(scenario['line'])
self.assertEqual(scenario['msg'], e.exception.message)
def _test_copy(self, line, outvar_name):
self._call_copy(line)
self.assertTrue(outvar_name in self.magics.shell.user_ns)
resp = self.magics.shell.user_ns[outvar_name]
self.assertEqual({}, resp.headers)
self.assertEqual(200, resp.status)
# sanity, no body
self.assertEqual('', resp.content)
def test_copy(self):
outvar_name = 'a1234'
line = ('--input path:/c/o --output path:/c/o '
'--storlet a.b -o %s' % outvar_name)
self._test_copy(line, outvar_name)
def test_copy_stdout_with_input(self):
params_name = 'params'
outvar_name = 'a1234'
line = ('--input path:/c/o --output path:/c/o '
'--storlet a.b -o %s -i %s' % (outvar_name, params_name))
self.magics.shell.register(params_name, {'a': 'b'})
self._test_copy(line, outvar_name)
def test_copy_stdout_with_input_error(self):
params_name = 'params'
outvar_name = 'a1234'
line = ('--input path:/c/o --output path:/c/o '
'--storlet a.b -o %s -i %s' % (outvar_name, params_name))
with self.assertRaises(KeyError):
self._test_copy(line, outvar_name)
class TestStorletMagicPut(BaseTestIpythonExtension, unittest.TestCase):
def setUp(self):
super(TestStorletMagicPut, self).setUp()
self.fake_connection = FakeConnection(201)
def _call_put(self, line):
self._call_line(self.magics.put, line)
def test_put_invalid_args(self):
scenarios = [
{
'line': '--input /c/o --storlet a.b --output path:/c/o',
'exception': UsageError,
'msg': '-o option is mandatory for the invocation'
}, {
'line': ('--input /c/o --storlet a.b -o 1234 '
'--output path:/c/o'),
'exception': UsageError,
'msg': ('The output variable name must be a valid prefix '
'of a python variable, that is, start with a '
'letter')
}, {
'line': '--input /c/o -o a1234 --output path:/c/o',
'exception': UsageError,
'msg': '--storlet option is mandatory for the invocation'
}, {
'line': '--storlet a.b -o a1234 --output path:/c/o',
'exception': UsageError,
'msg': '--input option is mandatory for the invocation'
}, {
'line': ('--input /c/o --storlet a.b -o a1234 '
'--output path/c/o'),
'exception': UsageError,
'msg': ('swift object path must have the format: '
'"path:/<container>/<object>"')
}, {
'line': ('--input path:c/ --storlet a.b -o a1234 '
'--output path:/c/o'),
'exception': UsageError,
'msg': ('--input argument must be a full path')
}]
for scenario in scenarios:
with self.assertRaises(UsageError) as e:
self._call_put(scenario['line'])
self.assertEqual(scenario['msg'], e.exception.message)
def _test_put(self, line, outvar_name):
open_name = '%s.open' % 'storlets.tools.extensions.ipython'
with mock.patch(open_name, create=True) as mock_open:
mock_open.return_value = mock.MagicMock(spec=file)
self._call_put(line)
self.assertTrue(outvar_name in self.magics.shell.user_ns)
resp = self.magics.shell.user_ns[outvar_name]
self.assertEqual({}, resp.headers)
self.assertEqual(201, resp.status)
# sanity, no body
self.assertEqual('', resp.content)
def test_put(self):
outvar_name = 'a1234'
line = ('--input /c/o --storlet a.b '
'--output path:a/b -o %s' % outvar_name)
self._test_put(line, outvar_name)
def test_put_stdout_with_input(self):
params_name = 'params'
outvar_name = 'a1234'
line = ('--input /c/o --storlet a.b -o %s -i %s '
'--output path:a/b' % (outvar_name, params_name))
self.magics.shell.register(params_name, {'a': 'b'})
self._test_put(line, outvar_name)
def test_put_stdout_with_input_error(self):
params_name = 'params'
outvar_name = 'a1234'
line = ('--input /c/o --storlet a.b -o %s -i %s '
'--output path:a/b' % (outvar_name, params_name))
with self.assertRaises(KeyError):
self._test_put(line, outvar_name)
if __name__ == '__main__':
unittest.main()