Middleware and Compute Engine for an OpenStack Swift compute framework that runs compute within a Swift cluster
Go to file
Takashi Kajinami 207f80b73c Remove unused constant
Change-Id: I13d013af686382716a943b4bc1c23157f4ff48b3
2024-02-11 00:08:30 +09:00
devstack Remove unused STORLETS_DOCKER_SWIFT_*_ID 2024-02-01 21:41:21 +09:00
doc Remove unused .unittests 2024-02-02 03:14:18 +09:00
etc Run codespell to fix typos 2024-01-30 19:09:27 +09:00
playbooks/storlets-functional Replace nose to nose2 2023-02-14 08:16:58 +00:00
releasenotes reno: Update master for unmaintained/yoga 2024-02-05 16:55:19 +00:00
src Run codespell to fix typos 2024-01-30 19:09:27 +09:00
storlets Remove unused constant 2024-02-11 00:08:30 +09:00
StorletSamples Run codespell to fix typos 2024-01-30 19:09:27 +09:00
tests Run codespell to fix typos 2024-01-30 19:09:27 +09:00
.coveragerc coveragerc: Remove non-existent path 2024-01-25 23:57:26 +09:00
.functests Replace nose to nose2 2023-02-14 08:16:58 +00:00
.gitignore Get rid of SUID binary to restart docker containers 2021-08-19 09:17:07 +00:00
.gitreview OpenDev Migration Patch 2019-04-19 19:47:54 +00:00
.mailmap Add mailmap entries for Takashi Kajinami 2024-01-02 18:38:03 +09:00
.stestr.conf Switch to use stestr for unit test 2018-07-20 13:59:22 +07:00
.zuul.yaml Skip functional tests for change in .gitignore 2024-02-02 03:12:09 +09:00
AUTHORS Update IRC network to OFTC 2021-06-01 00:40:19 +09:00
bindep.txt Drop explicit install of uwsgi-plugin-python3 2024-01-08 11:11:02 +09:00
build.xml Remove deploy targets 2024-01-07 15:05:03 +09:00
CHANGELOG Run codespell to fix typos 2024-01-30 19:09:27 +09:00
CONTRIBUTING.rst Replace github by opendev 2022-05-04 21:54:14 +09:00
HACKING.rst Update the documention link for doc migration 2017-10-16 15:03:04 +08:00
install_libs.sh Offload entry point and command from image 2023-12-13 15:32:09 +09:00
LICENSE Initial Cookiecutter Commit. 2015-08-09 18:08:41 +03:00
nose2.cfg Replace nose to nose2 2023-02-14 08:16:58 +00:00
README.rst Update repository structure 2024-02-02 02:47:49 +09:00
requirements.txt Limit cpu and memory used by each storlet container 2024-01-28 21:56:35 +09:00
s2aio.sh Use devstack functions to manage swift/keystone services 2024-01-08 13:42:35 +09:00
setup.cfg Run codespell to fix typos 2024-01-30 19:09:27 +09:00
setup.py Fix python shebang 2023-10-18 12:37:12 +09:00
test-requirements.txt Remove note about old pip's behavior 2024-01-08 23:50:43 +09:00
tox.ini Fix redundant default posargs for unit tests 2024-02-01 19:01:55 +00:00

Storlets

Storlets extend Swift with the ability to run user defined computations - called storlets - near the data in a secure and isolated manner. A storlet is a compiled and packaged code (e.g. a .jar file) that can be uploaded to Swift as any other object. Once uploaded the storlet can be invoked over data objects in Swift. The Storlets API is documented at "Storlets API v1".

The project started off as an IBM research project, and was open sourced by IBM in 2014.

  • The research leading to the development of this code received funding from the European Community's Seventh Framework Programme (FP7/2007-2013) under the grant agreements for the CASPAR, ENSURE and VISION Cloud projects.
  • Various stages and different aspects of the development of this code received funding from the following European Community's Framework Programme:
    • Seventh Framework Programme (FP7/2007-2013) under the grant agreements for the ForgetIT project, where the code is used for offloading digital preservation functionalities to the storage.
    • Seventh Framework Programme (FP7/2007-2013) under the grant agreements for COSMOS project, where the code is used for analysis of IoT data.
    • Seventh Framework Programme (FP7/2007-2013) under the grant agreements for FI-CORE project where the code is integrated with a holistic cloud deployment solution, and from
    • Horizon 2020 (H2020/2014-2020) under the grant agreement for the IOStack project where the code is used as a backend implementing Storage policies and is used for analytics

Docs

The storlerts documentation is auto-generated after every commit and available online at https://docs.openstack.org/storlets/latest/

Getting Started for Users

The fastest way to get started is "S2AIO - Swift Storlets All In One".

For Engine Developers

Getting Started

The best way to get started is following this guide: "Installing a Development Environment".

Tests

There are two types of tests included in the Storlets repo.

  1. Unit tests
  2. Functional tests

Unit tests, are, well, unit tests... The functional tests are black box tests validating end-to-end scenarios using various storlets, including faulty ones. For more information please refer to the: "Development and Testing Guide".

Repository Structure

  • doc/source/: Documentation
  • etc/: Sample config files
  • storlets/: Python codes
    • agent/: Python code for Docker side agents
      • common/: An agent for storlets process management
      • daemon/: An agent for execution of python applications
      • daemon_factory/: Pyth
    • gateway/: Run time loadable code for managing storlets execution
    • sbus/: A Java implementation of the SBUS communication protocol
    • swift_middleware/: Swift middleware dealing with storlet invocation requests
  • StorletSamples/: Storlets examples, used for functional testing
  • src/: C and Java codes
    • c/: All codes
      • sbus/: A core implementation of the SBUS protocol, which is used for passing fsd between the middleware and container
    • java/: Java codes
      • SBus:/ A Java implementation of the SBUS communication protocol
      • SCommon/: A Java library required for storlets development
      • SDaemon/: A generic Java daemon for loading storlets at runtime
  • tests/: Unit and functional tests
  • tools/: Various cluster config dependent tools for automatic and manual testing

For Storlets Developers

Currently, storlets can be developed in Java only. To get started, follow: "S2AIO - Swift Storlets All In One".

The write and deploy a storlet, follow: "Writing and deploying storlets".