c8087fa0a4
Added paragraph to the NFS-Ganesha docs with a brief explanation of the role of ``dbus`` messaging when doing dynamic updates of exports. Key point is that ``dbus-send`` and NFS-Ganesha must be in the same namespace even if NFS-ganesha runs in a container. Close-bug: #1883961 Change-Id: I57b916b16d07a8373143f396b42ea34bf80330a6
344 lines
13 KiB
ReStructuredText
344 lines
13 KiB
ReStructuredText
..
|
|
Copyright 2015 Red Hat, Inc.
|
|
All Rights Reserved.
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License"); you may
|
|
not use this file except in compliance with the License. You may obtain
|
|
a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
|
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
|
License for the specific language governing permissions and limitations
|
|
under the License.
|
|
|
|
Ganesha Library
|
|
===============
|
|
|
|
The Ganesha Library provides base classes that can be used by drivers to
|
|
provision shares via NFS (NFSv3 and NFSv4), utilizing the NFS-Ganesha NFS
|
|
server.
|
|
|
|
Supported operations
|
|
--------------------
|
|
|
|
- Allow NFS Share access
|
|
|
|
- Only IP access type is supported.
|
|
|
|
- Deny NFS Share access
|
|
|
|
Supported manila drivers
|
|
------------------------
|
|
|
|
- CephFS driver uses ``ganesha.GaneshaNASHelper2`` library class
|
|
|
|
- GlusterFS driver uses ``ganesha.GaneshaNASHelper`` library class
|
|
|
|
Requirements
|
|
------------
|
|
|
|
- Preferred:
|
|
|
|
`NFS-Ganesha <https://github.com/nfs-ganesha/nfs-ganesha/wiki>`_ v2.4 or
|
|
later, which allows dynamic update of access rules. Use with manila's
|
|
``ganesha.GaneshaNASHelper2`` class as described later in
|
|
:ref:`using_ganesha_library`.
|
|
|
|
(or)
|
|
|
|
`NFS-Ganesha <https://github.com/nfs-ganesha/nfs-ganesha/wiki>`_ v2.5.4 or
|
|
later that allows dynamic update of access rules, and can make use of highly
|
|
available Ceph RADOS (distributed object storage) as its shared storage for
|
|
NFS client recovery data, and exports. Use with Ceph v12.2.2 or later, and
|
|
``ganesha.GaneshaNASHelper2`` library class in manila Queens release or
|
|
later.
|
|
|
|
- For use with limitations documented in :ref:`ganesha_known_issues`:
|
|
|
|
`NFS-Ganesha <https://github.com/nfs-ganesha/nfs-ganesha/wiki>`_ v2.1 to
|
|
v2.3. Use with manila's ``ganesha.GaneshaNASHelper`` class as described later
|
|
in :ref:`using_ganesha_library`.
|
|
|
|
NFS-Ganesha configuration
|
|
-------------------------
|
|
|
|
The library has just modest requirements against general NFS-Ganesha (in the
|
|
following: Ganesha) configuration; a best effort was made to remain agnostic
|
|
towards it as much as possible. This section describes the few requirements.
|
|
|
|
Note that Ganesha's concept of storage backend modules is called FSAL ("File
|
|
System Abstraction Layer"). The FSAL the driver intends to leverage needs to be
|
|
enabled in Ganesha config.
|
|
|
|
Beyond that (with default manila config) the following line is needed to be
|
|
present in the Ganesha config file (that defaults to
|
|
/etc/ganesha/ganesha.conf):
|
|
|
|
``%include /etc/ganesha/export.d/INDEX.conf``
|
|
|
|
The above paths can be customized through manila configuration as follows:
|
|
|
|
- `ganesha_config_dir` = toplevel directory for Ganesha configuration,
|
|
defaults to /etc/ganesha
|
|
- `ganesha_config_path` = location of the Ganesha config file, defaults
|
|
to ganesha.conf in `ganesha_config_dir`
|
|
- `ganesha_export_dir` = directory where manila generated config bits are
|
|
stored, defaults to `export.d` in `ganesha_config_dir`. The following
|
|
line is required to be included (with value expanded) in the Ganesha
|
|
config file (at `ganesha_config_path`):
|
|
|
|
``%include <ganesha_export_dir>/INDEX.conf``
|
|
|
|
In versions 2.5.4 or later, Ganesha can store NFS client recovery data in
|
|
Ceph RADOS, and also read exports stored in Ceph RADOS. These features are
|
|
useful to make Ganesha server that has access to a Ceph (luminous or later)
|
|
storage backend, highly available. The Ganesha library class
|
|
`GaneshaNASHelper2` (in manila Queens or later) allows you to store Ganesha
|
|
exports directly in a shared storage, RADOS objects, by setting the following
|
|
manila config options in the driver section:
|
|
|
|
- `ganesha_rados_store_enable` = 'True' to persist Ganesha exports and export
|
|
counter in Ceph RADOS objects
|
|
- `ganesha_rados_store_pool_name` = name of the Ceph RADOS pool to store
|
|
Ganesha exports and export counter objects
|
|
- `ganesha_rados_export_index` = name of the Ceph RADOS object used to store
|
|
a list of export RADOS object URLs (defaults to 'ganesha-export-index')
|
|
|
|
Check out the `cephfs_driver` documentation for an example driver section
|
|
that uses these options.
|
|
|
|
To allow Ganesha to read from RADOS objects add the below code block in
|
|
ganesha's configuration file, substituting values per your setup.
|
|
|
|
.. code-block:: console
|
|
|
|
# To read exports from RADOS objects
|
|
RADOS_URLS {
|
|
ceph_conf = "/etc/ceph/ceph.conf";
|
|
userid = "admin";
|
|
}
|
|
# Replace with actual pool name, and export index object
|
|
%url rados://<ganesha_rados_store_pool_name>/<ganesha_rados_export_index>
|
|
# To store client recovery data in the same RADOS pool
|
|
NFSv4 {
|
|
RecoveryBackend = "rados_kv";
|
|
}
|
|
RADOS_KV {
|
|
ceph_conf = "/etc/ceph/ceph.conf";
|
|
userid = "admin";
|
|
# Replace with actual pool name
|
|
pool = <ganesha_rados_store_pool_name>;
|
|
}
|
|
|
|
For a fresh setup, make sure to create the Ganesha export index object as an
|
|
empty object before starting the Ganesha server.
|
|
|
|
.. code-block:: console
|
|
|
|
echo | sudo rados -p ${GANESHA_RADOS_STORE_POOL_NAME} put ganesha-export-index -
|
|
|
|
Further Ganesha related manila configuration
|
|
--------------------------------------------
|
|
|
|
There are further Ganesha related options in manila (which affect the
|
|
behavior of Ganesha, but do not affect how to set up the Ganesha service
|
|
itself).
|
|
|
|
These are:
|
|
|
|
- `ganesha_service_name` = name of the system service representing Ganesha,
|
|
defaults to ganesha.nfsd
|
|
- `ganesha_db_path` = location of on-disk database storing permanent Ganesha
|
|
state, e.g. an export ID counter to generate export IDs for shares
|
|
|
|
(or)
|
|
|
|
When `ganesha_rados_store_enabled` is set to True, the ganesha export
|
|
counter is stored in a Ceph RADOS object instead of in a SQLite database
|
|
local to the manila driver. The counter can be optionally configured with,
|
|
`ganesha_rados_export_counter` = name of the Ceph RADOS object used as the
|
|
Ganesha export counter (defaults to 'ganesha-export-counter')
|
|
|
|
- `ganesha_export_template_dir` = directory from where Ganesha loads
|
|
export customizations (cf. "Customizing Ganesha exports").
|
|
|
|
.. _using_ganesha_library:
|
|
|
|
Using Ganesha Library in drivers
|
|
--------------------------------
|
|
|
|
A driver that wants to use the Ganesha Library has to inherit
|
|
from ``driver.GaneshaMixin``.
|
|
|
|
The driver has to contain a subclass of ``ganesha.GaneshaNASHelper2``,
|
|
instantiate it along with the driver instance and delegate
|
|
``update_access`` method to it (when appropriate, i.e., when ``access_proto``
|
|
is NFS).
|
|
|
|
.. note::
|
|
|
|
You can also subclass ``ganesha.GaneshaNASHelper``. It works with
|
|
NFS-Ganesha v2.1 to v2.3 that doesn't support dynamic update of exports.
|
|
To update access rules without having to restart NFS-Ganesha server, the
|
|
class manipulates exports created per share access rule (rather than per
|
|
share) introducing limitations documented in :ref:`ganesha_known_issues`.
|
|
|
|
|
|
In the following we explain what has to be implemented by the
|
|
``ganesha.GaneshaNASHelper2`` subclass (to which we refer as "helper
|
|
class").
|
|
|
|
Ganesha exports are described by so-called *Ganesha export blocks*
|
|
(introduced in the 2.* release series), that is, snippets of Ganesha
|
|
config specifying key-pair values.
|
|
|
|
The Ganesha Library generates sane default export blocks for the
|
|
exports it manages, with one thing left blank, the so-called *FSAL
|
|
subblock*. The helper class has to implement the ``_fsal_hook``
|
|
method which returns the FSAL subblock (in Python represented as
|
|
a dict with string keys and values). It has one mandatory key,
|
|
``Name``, to which the value should be the name of the FSAL
|
|
(eg.: ``{"Name": "CEPH"}``). Further content of it is
|
|
optional and FSAL specific.
|
|
|
|
Customizing Ganesha exports
|
|
---------------------------
|
|
|
|
As noted, the Ganesha Library provides sane general defaults.
|
|
|
|
However, the driver is allowed to:
|
|
|
|
- customize defaults
|
|
- allow users to customize exports
|
|
|
|
The config format for Ganesha Library is called *export block
|
|
template*. They are syntactically either Ganesha export blocks,
|
|
(please consult the Ganesha documentation about the format),
|
|
or isomorphic JSON (as Ganesha export blocks are by-and-large
|
|
equivalent to arrayless JSON), with two special placeholders
|
|
for values: ``@config`` and ``@runtime``. ``@config`` means a
|
|
value that shall be filled from manila config, and ``@runtime``
|
|
means a value that's filled at runtime with dynamic data.
|
|
|
|
As an example, we show the library's defaults in JSON format
|
|
(also valid Python literal):
|
|
|
|
::
|
|
|
|
{
|
|
"EXPORT": {
|
|
"Export_Id": "@runtime",
|
|
"Path": "@runtime",
|
|
"FSAL": {
|
|
"Name": "@config"
|
|
},
|
|
"Pseudo": "@runtime",
|
|
"SecType": "sys",
|
|
"Tag": "@runtime",
|
|
"CLIENT": {
|
|
"Clients": "@runtime",
|
|
"Access_Type": "RW"
|
|
},
|
|
"Squash": "None"
|
|
}
|
|
}
|
|
|
|
The Ganesha Library takes these values from
|
|
|
|
*manila/share/drivers/ganesha/conf/00-base-export-template.conf*
|
|
|
|
where the same data is stored in Ganesha conf format (also supplied with
|
|
comments).
|
|
|
|
For customization, the driver has to extend the ``_default_config_hook``
|
|
method as follows:
|
|
|
|
- take the result of the super method (a dict representing an export
|
|
block template)
|
|
- set up another export block dict that include your custom values,
|
|
either by
|
|
|
|
- using a predefined export block dict stored in code
|
|
- loading a predefined export block from the manila source tree
|
|
- loading an export block from an user exposed location (to allow
|
|
user configuration)
|
|
|
|
- merge the two export block dict using the ``ganesha_utils.patch``
|
|
method
|
|
- return the result
|
|
|
|
With respect to *loading export blocks*, that can be done through the
|
|
utility method ``_load_conf_dir``.
|
|
|
|
Known Restrictions
|
|
------------------
|
|
|
|
- The library does not support network segmented multi-tenancy model but
|
|
instead works over a flat network, where the tenants share a network.
|
|
|
|
.. _ganesha_known_issues:
|
|
|
|
Known Issues
|
|
------------
|
|
|
|
Following issues concern only users of `ganesha.GaneshaNASHelper` class that
|
|
works with NFS-Ganesha v2.1 to v2.3.
|
|
|
|
- The export location for shares of a driver that uses the Ganesha Library
|
|
will be of the format ``<ganesha-server>:/share-<share-id>``. However,
|
|
this is incomplete information, because it pertains only to NFSv3
|
|
access, which is partially broken. NFSv4 mounts work well but the
|
|
actual NFSv4 export paths differ from the above. In detail:
|
|
|
|
- The export location is usable only for NFSv3 mounts.
|
|
- The export location works only for the first access
|
|
rule that's added for the given share. Tenants that
|
|
should be allowed to access according to a further
|
|
access rule will be refused (cf.
|
|
https://bugs.launchpad.net/manila/+bug/1513061).
|
|
- The share is, however, exported through NFSv4, just
|
|
on paths that differ from the one indicated by
|
|
the export location, namely at:
|
|
``<ganesha-server>:/share-<share-id>--<access-id>``,
|
|
where ``<access-id>`` ranges over the ID-s of access
|
|
rules of the share (and the export with ``<access-id>``
|
|
is accessible according to the access rule of that ID).
|
|
- NFSv4 access also works with pseudofs. That is, the
|
|
tenant can do a v4 mount of``<ganesha-server>:/`` and
|
|
access the shares allowed for her at the respective
|
|
``share-<share-id>--<access-id>`` subdirectories.
|
|
|
|
Deployment considerations
|
|
-------------------------
|
|
|
|
When using `NFS-Ganesha`_ v2.4 or later and manila's
|
|
``ganesha.GaneshaNASHelper2`` class, dynamic export of
|
|
access rules is implemented by using the `dbus-send`_ command to
|
|
signal `NFS-Ganesha`_ to update its exports.
|
|
The `dbus-send`_ command is executed on the host where `NFS-Ganesha`_
|
|
runs. This may be the same host where the :term:`manila-share`
|
|
service runs, or it may be remote to :term:`manila-share` depending on
|
|
how the relevant driver has been configured. Either way, the `dbus-send`_
|
|
command and `NFS-Ganesha`_ must be able to communicate over an *abstract
|
|
socket* and *must be in the same namespace*. Consequently, if you deploy
|
|
`NFS-Ganesha`_ in a container you likely should run the container in
|
|
the host namespace (e.g. 'docker run --net=host ...') rather than in its
|
|
own network namespace. For details, see this
|
|
`article <https://stackoverflow.com/questions/38455283/docker-containers-share-unix-abstract-socket-or-dbus>`_.
|
|
|
|
The :mod:`manila.share.drivers.ganesha` Module
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. automodule:: manila.share.drivers.ganesha
|
|
:noindex:
|
|
:members:
|
|
:undoc-members:
|
|
:show-inheritance:
|
|
|
|
.. _NFS-Ganesha: https://github.com/nfs-ganesha/nfs-ganesha/wiki
|
|
.. _dbus-send: https://dbus.freedesktop.org/doc/dbus-send.1.html
|
|
|