This change allows a deployer to selectively enable or disable the
template engine. This will allow folks to override the contents of
a configuration file via the `config_overrides` option but not render
any jinja related content within the file, except for what is provided
by the `config_overrides` option.
Change-Id: If226b4814a88b3bd7968b5e2ef64474710665ae0
Signed-off-by: Kevin Carter <kecarter@redhat.com>
This change adds traps for the FileNotFound exception which is used instead
of OSError when a file is not found. This will ensure that that fall back
exception handeling works properly in py2 and py3.
The remote_src flag was not being respected with the new fallback options,
this change adds a check for remote_src so it works in normal and fallback
mode.
Change-Id: I136dd1f3220d3b004d5275a9fa15edea53d58f6b
Signed-off-by: Kevin Carter <kecarter@redhat.com>
This change will now fall back to using the ansible core module
`stat` to stat a file when the local stat function fails for a
permissions error. This is being done because there is an issue
with action plugins not respecting "become: true" on local files
being written over a local connection.
Change-Id: I646f0688f1f4bf3cddaef9114dac51c2a807b25b
Signed-off-by: Kevin Carter <kecarter@redhat.com>
Sometimes there might be a key in configfile, that should be empty but
defined correctly. This patch fixes such behaviour and adds testing.
Change-Id: I693bc3a2aade7015d0465583001962df4ba53443
This change will allow config_template to parse list only files
and extend or replace the content of the list based on the
selected module options.
Change-Id: I2a24033b0323bcc25bd1b50ffb4034441ab2f468
Signed-off-by: Kevin Carter <kecarter@redhat.com>
The config_template comment parser will now respect all comments in
INI files, as they are written. This will ensure spacing, paragraphs,
and other comments bits that may be in an INI file remain intact,
even when overriding options and sections. With this feature we'll
now be able to insert options in OSLO config generated files without
truncating or making a mess of the file structure.
This is an internal enhancement and requires no change from the
operator or from within any ansible task. To ensure enhanced comments
are working, new tests have been added which will run though all of
the `config_template` INI file functions using a mock service file.
Existing tests for the old comment structure has been removed.
This includes tests that were running redundant tasks for file
diffs or expected the only style, merged, comment layout. All of the
tests have been broken out into descriptive task files. This was
largely done for readability. It was difficult to see what tests
we had and how I needed to extend them to test the enhanced comments
functionality given the INI type tests were all thoughout the
`tests/test-common-tasks.yml`. Now that the files have been broken
out developers will easily be able to audit our tests making it
simple to extend.
Change-Id: Ia6cdc215c35fa9ac45b718c211616a9887a74e37
Signed-off-by: Kevin Carter <kecarter@redhat.com>
The config template action plugin assumes that the local node is
supplying a configuration file that will be used to copy content
to a target node. This change gives the module the ability to
change configuration files on target nodes using all remote
sources. The new boolean option, `remote_src` has been added to
enable or disable this functionality. When enabled, the module
will retrieve content from the target node and load it as
"user_content". By pre-loading it in the user content the
module will function normally without any major structural
changes or performance impact.
Change-Id: Id9c7e16fb935c2da0b32b7cf53449f68bd1e9c89
Signed-off-by: Kevin Carter <kecarter@redhat.com>
Added a flag called yml_multilines which when used will not split on
lines that have a newline UNLESS they have a comma.
Change-Id: I90364403e215a67b320dfc7e67a85d47c774e634
Partial-Bug: #1819974
Python 2.7 ConfigParser does not accept whitespace before comments.
To keep config files produced by ansible-config_temple parsable
we will strip out the whitespace preceding comments.
Change-Id: I39c88578d0a9e42fded7410d7eb4ec8bd415550c
Related-Bug: #1755821
This commit adds a check for "changed" in the return from the copy
module. If the value is not possitive `rc['changed']` will be set
with the constant "changed" from within the action plugin.
Change-Id: Id65e6e57d174c4c25d99836b09ad049302d4e42c
Signed-off-by: cloudnull <kevin@cloudnull.com>
Compare dict vars of before and after configuration to determine whether
the config keys or values have changed so a configuration file will not
be incorrectly marked as changed when only the ordering has changed.
Set diff return variable to a dict of changes applied.
Change-Id: Ie67119b1420936c8ed89f8338ea9dce4c47e185c
The unicode() built-in does not exist under Python 3 so use
six.text_type, which is set correctly to str or unicode, instead.
Change-Id: I9aa5df72b2248f9bd32e1c49118d0eb889a95691
In Ansible 2.6 the copy module now recognizes the "_original_basename"
option and not the "original_basename" option. In order for this module
to work with versions less than 2.6.0 and 2.6.0+ we need to allow an if
statement based on the version of Ansible being used.
Change-Id: I3f7a12dec77ea6e8e3c9fcae6ce9c162df57e50c
In ansible 2.5 the way files are generated changes meaning that --diff
mode will stop working, and generate an error instead. Pending a better
solution, we should remove --diff mode.
Change-Id: I0d9b7baf08be00b978b8198f95db8962e7fa7787
Due to the ordering of the vars passed to the _get_diff_data mode,
--diff mode would only work with --check activated, otherwise the file
used didn't exist and you would get an error similar to:
https://github.com/ceph/ceph-ansible/issues/2742
This PR allows this to work by configuring the diff data before running
the copy module, so the file exists.
Additionally, we should run with --diff mode enabled by default on tests
for this repository to ensure any functional issues with diff mode are
caught.
Change-Id: I020a181731c3ee87f04933f8c62213ea12cfa3a4
The default section in an ini file will always be 'DEFAULT', but for some
configuration files you may want to be able to set this to ensure
another section is at the top. 'global' for example, when working with
fio.
This PR adds the ability to set the section to be at the top, but will
default to 'DEFAULT' so no change will happen to existing tasks running
config_template.
Change-Id: Ifbc0b91aef46f3c2d98e73fdc9ab888244550bab
unicode is nolonger a type in py3 - instead the str type is used for
unicode, and the bytes type would cover the str type from py2.
To avoid having to version the code, we should try to use unicode,
except the NameError and continue as though it is py3 if that fails.
Additionally, this patch adds a test that will fail if you revert the
config_template.py back to it's original in py3.
Closes-bug: 1763422
Change-Id: Ifda972caada27ade2d80f77b3df70568406226ff
When using config_template with a MultiStringOp the order was subject to
change because of the use of set theory. This change converts the use of
sets to tuples which allow us to retain the ability to override options
without losing the ordering of keys.
Change-Id: Id0bba86629a33ce74528c0bd51b11183f5315d86
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
(cherry picked from commit 662a2805c0dce43a1efec31a85e09056207ec3d2)
As part of the ceph-ansible project, this change was put in:
ec042219e6
This was to handle the case where the config would change, initiating a
restart handler, even though no actual change happened.
We can juse use sorted() since we are passing back a list with .items()
which can then be sorted.
The above change didn't include a change to ensure the order of the
sections, which meant that although the individual items were sorted the
sections could change order causing unnecessary changes.
Additionally, this adds a test to retemplate the file 3 times, with a
lot more sections to ensure the order remains the same, this will fail
when the config template changes even though no changes have happened.
Change-Id: I7c7cd9ef1dd767a2a5d754ac4663cf8655dd60c2
(cherry-picked from plugins repository)
Gate tests will fail until we fix pep8 for config_template.py
This PR fixes the config_template module for pep8 compliance,
follows the latest PTI for docs compliance,
and re-enables the linters.
Change-Id: Ic288b462a53ac45983f47f29a7d5d8d48d0b9c0b
Co-Authored-By: Jean-Philippe Evrard <jean-philippe@evrard.me>
config_template currently strips comments out from the INI files. This
commit adds support for comments.
The default ConfigParser supports reading comments but it doesn't writes
them back. There are other libraries that have better support for
comments, however, adopting another library would require a bigger
refactor of this code.
Change-Id: I376be69cd2cf34dc147f9ade716791125a424e2c
Ansible v1.9 was last used by OSA in Mitaka. The extra code to keep the
config_template and py_pkgs plugins compatible should no longer be
needed.
Change-Id: I14b32af1dc9bfe8b09c52f0205afaaad9139b58c
Strings are used where they were encoded to bytes and cast back to
strings before. The StringIO module is imported to retain backwards
compatibility with Python 2.7. Also, the convert_2_string function is
removed and its usage replaced with Ansible's to_text function.
Change-Id: I396d96b8047e653f4526350375abc1f5f0dcd756
Fixes-Bug: 1711347
iteritems is a py2 callable and will break when using py3.
Change-Id: Icea444baea8f3597cb8dd24e3d61e9ed0926cdf5
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Using AnsibleDumper adds support for dumping the internal ansible
data types that are used in hostvars.
Fixes-Bug: #1707265
Change-Id: Ie59bbef31306cd3e99aa827db875922e17084e83
It is sometimes useful to tell ConfigTemplateParser to write
out options that are valueless and not suffixed with '=' or ':',
such as when overriding a my.cnf.
This commit adds the 'ignore_none_type' attribute to the config_template
module. If this attribute is set to false, then valueless options will be
written out to the resultant INI file as-is, without the '=' or ':' suffix.
Change-Id: I5c88b2019c01b44193a5d0df9299ecce6de52f01
Partial-Bug: #1693234
When users specificy the 'content' attribute when
using the config_template module, it's possible that
this input can be a dict. This commit handles
dictionary input, and dumps it accordingly based
on 'config_type'
Change-Id: I1f12810ad7ce5746f8938ec9d608c532ba530d58
The default py yaml `safe_dumps` keeps to the yaml spec which states
incremental indentations are not required. Sadly, without the
indentations some consumers of yaml configurations files will break.
This commit adds an incremental indentation dumper to cater for all
yaml configuration consumers.
Change-Id: Ifa5bcc6ff9f0a9cb5e619e62d0356f5d6887ca6e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
When using content: the config_template module will create a
temporary file. This patch ensures a deterministic path to the
temp file is used, and removes it at the end of the module
execution.
Change-Id: I3b3354178ba1a2b9c95decb3d3e0c9a1c2689056
Copy module can take content argument, I think we should introduce this
feature to config_template. It brings the possibility to use lookups for
inline passing of content to template.
Example, we could use:
config_template:
content: "{{ lookup('file',<file>) }}" to replace current behavior
or:
config_template:
content: "{{ lookup('url','<url>',wantlist=True) | join ('\n') }}"
to bring inline templating of external sources.
Change-Id: Id5b2743d309f0313603afbbf84279ce0b1e49cfb
Signed-off-by: Jean-Philippe Evrard <jean-philippe.evrard@rackspace.co.uk>
The config_template is using the old ansible objects, this should
make it work with the newest standards.
Change-Id: I9159be497c201ca6c2f558336843be5175f0ee2c
Closes-Bug: 1640458
Ansible has changed the variable ansible_ssh_user to ansible_user
this change adds support for the option ansible_user while
maintaining compatibility with the legacy option.
Change-Id: I032a3562ccebbbf8f7248bfbec1e2c59b452587b
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The first_available_file option on tasks has been deprecated in
Ansible 2, and will be removed in 2.2. Remove its use in the
config_template plugin similarly to its removal from the upstream
template plugin [0].
[0] https://github.com/ansible/ansible/pull/17643
Change-Id: Icd4f70706979a3f59dd93f116d1c5ef96d66edbe
By default a list item in a JSON or YAML format will extend if a its already
defined in the target template and a config_override using a list is being set
for the existing "key".
This change allows that functionality to be toggled on or off. The boolean
option to enable list extension is ``list_extend``.
To maintain the API this feature will remain True by default.
Co-Authored-By: Travis Truman <travis_truman@cable.comcast.com>
Change-Id: I97e06aef2cc778f048f3d6863fe61d10eddb8602
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change allows keys in INI format to be any case.
The default ConfigParse module sets this to be lower
however in some cases keys are needed to be upper and/or
mixed.
Change-Id: I4e0dedb1b73ee596929bd425af6b0aaefd3a6c27
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
These changes address 2 issues:
* The config template plugin in v1 mode was not respecting a list
of integers and would cause a stacktrace because the ConfigParse
module expects everything written to be a string.
* The config template plugin in v2 mode was not properly handling
the multi-string-op type due to an oversight in the ``_option_write``
method.
With these updates the config template is now able to better support
all of the potential options that could be thrown at it when writing
an INI config file.
These issues we're discovered while working with the ceph-ansible
project and a mirroring PR has been created here:
https://github.com/ceph/ceph-ansible/pull/707
Change-Id: I051096abae883ac81195d4538140566b69f8996c
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change updates the config_template plugin to load
the version of the config_template needed based on the
detected version of ansible running. Note, Once the
upstream change merges for ansible 2 [0] we should
drop the ansible 2 version of this plugin.
[0] - https://github.com/ansible/ansible/pull/12555
Change-Id: Ifbdde0f30ee2bd8025ede07c86234527ba9891db
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change adds the MultiStrOps variaable type to the config
template parsing as well as supports multi string options when
passing configuration overrides.
This is made prossible by using the set type for options found within
config overrides and creating a custom dictionary class that allows
for multiple keys to be stored as a set. When the config_template
encounters a set type it will process and reder value as a
MultiStrOps.
Set types are defined in yaml via the "?" entry.
Example Overrides:
things:
- 1
- 2
multistrops_things:
? a
? b
Example Rendered Config:
things = 1,2
multistrops_things = a
multistrops_things = b
Change-Id: I2193ea2eb7f839c3151c2c96f9dfe86f141e5a15
Closes-Bug: #1542513
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The standard Ansible names for the plugins folders [1] are singular.
This patch renames our plural names to singular names so that when
the plugins are downloaded into a default location there is no further
configuration required.
[1] https://github.com/ansible/ansible/blob/devel/lib/ansible/constants.py
Change-Id: Ib965750f267d11287a6196e2544c3a014a3e2029
