Commit 5e61fbaf00c97 added a new --enabled-only option but that
doesn't work correctly because the xml_jobs_filtered variable isn't
filled with all the jobs in the end. So some jobs are not updated even
if the disabled status was not found. That's fixed now.
Change-Id: I915116cf875e9e20c10ca43175c924b9e4307cca
Rewrite YAML parser, YAML objects and parameters expansion logic to
enable better control over expansion logic.
Broken backward compatilibity:
* More agressive parameter expansion. This may lead to parameters
expanded in places where they were not expanded before.
* Top-level elements, which is not known to parser (such as 'job',
'view', 'project' etc), are now lead to parse failures.
Prepend them with underscore to be ignored by parser.
* Files included using '!include-raw:' elements and having formatting in
it's path ('lazy-loaded' in previous implementation) are now expanded
too.
Use '!include-raw-escape:' for them instead.
See changes in these tests for examples:
tests/yamlparser/job_fixtures/lazy-load-jobs-multi001.yaml
tests/yamlparser/job_fixtures/lazy-load-jobs-multi002.yaml
tests/yamlparser/job_fixtures/lazy-load-jobs001.yaml
* Parameters with template value using itself were substituted as is.
For example: "timer: '{timer}'" was expanded to "{timer}". Now it
leads to recursive parameter error.
See changes in this test for example:
tests/yamlparser/job_fixtures/parameter_name_reuse_default.*
->
tests/yamlparser/error_fixtures/parameter_name_reuse_default.*
* When job group includes a job which was never declared, it was just
ignored. Now it fails: job is missing.
See changes in this test for example:
tests/yamlparser/job_fixtures/job_group_includes_missing_job.*
->
tests/yamlparser/error_fixtures/job_group_includes_missing_job.*
Change-Id: Ief4e515f065a1b9e0f74fe06d7e94fa77d69f273
That option can be used when calling the update subcommand. If the
option is set, only jobs which are enabled will be updated.
That is useful of there are a lot of disabled jobs which are not
required.
Change-Id: Ie2f268e2356a3b7e0a334a331696ca81e09790aa
Resolves flake8 BLK related issues.
"BLK100 Black would make changes."
"BLK999 Unexpected exception: ..."
Change-Id: Ic6d4265e3d02b2f2a15ab851eaed98288c62730c
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
In the new version of PyYAML the API changed to be more explicit. Now
the default value for the Loader is None, see:
https://github.com/yaml/pyyaml/blob/5.1/lib3/yaml/__init__.py#L103
The load is unsafe. It's better to use safe_load function.
Change-Id: Ia1cd16f2ff970ca220a266c99b6554dd4254ebd9
Please review the following patch containing the code changes in
the repo. This patch is a transition patch and is the auto-generated
output of the python-black tool.
Change-Id: I2d2de71da8a105fb62b561899ae78441ddab4032
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Instead of printing the deprecation warning against every generated
config-xml file, print it only once at the beginning of the test
command call to not spam the console.
Change-Id: Ifdcdb1373d44ba6232ea7a5359662cc760353774
Signed-off-by: Thanh Ha <zxiiro@linux.com>
If the user does not have 'update' in their configuration
ensure we are properly defaulting to 'all' and not None.
Change-Id: I65ddc49066119db0947801203ac88cb69153d1b0
Signed-off-by: Thanh Ha <zxiiro@linux.com>
Add options --jobs-only and --views-only to job_builder
section in config file.
By default JJB updates both jobs and views. Some cases
non-admin users are not allowed to update views on Jenkins
which requires explicity using -j flag.
Allow users to set a config option 'update=jobs|views|all'
in the 'job_builder' section implicitly use --jobs-only or
--views-only and control these flags from the config file.
Note: CLI options takes precedence over the config file.
Change-Id: I2a94e5a2d671ccbfc505de2f19b578ecfef9e9d7
Co-Authored-By: Thanh Ha <zxiiro@linux.com>
Signed-off-by: Anil Belur <askb23@gmail.com>
Signed-off-by: Thanh Ha <zxiiro@linux.com>
Signed-off-by: Anil Belur <askb23@gmail.com>
Enable us to do `python -m jenkins_jobs ...`
which avoids the need to install executable in
user path.
Tests cli execution on each supported python version.
Uses default encoding workaround only on
python2, where is needed.
Change-Id: I4cd79fd51a8309d532e0e76723ecfbbda3e1ca6f
Signed-off-by: Sorin Sbarnea <ssbarnea@redhat.com>
- upgrades hacking to current version
- sorts new linting issues
- sorts bug with sys.reload on py3
Change-Id: I4a18abc93116667a2733e8aec619ac59ea73d630
Signed-off-by: Sorin Sbarnea <ssbarnea@redhat.com>
Printed job urls with unicode are now clickable
Avoids unicode decode errors in logging due on
systems, likely python27 systems where
default encoding is ascii.
Change-Id: I17041de05a4388c7b95a9ca04477043424458c67
Additional argument for the update subcommand to allow user to update
only jobs currently installed on the Jenkins server and managed by
this tool.
Change-Id: I4b0e997df01057f89896b8caa27dc00b5ca077df
Add a list subcommand that allows listing of all the jobs defined in
yaml configuration, or on the Jenkins master, with the option of taking
glob parameters to allow matching some job names.
Also supports the recursive and exclude path control options.
jenkins-jobs list 'wikimedia-fundraising*'
jenkins-jobs list -r -p configs/ 'wikimedia-fundraising*'
Co-Authored-By: Darragh Bailey <dbailey@hpe.com>
Co-Authored-By: Sorin Sbarnea <ssbarnea@redhat.com>
Change-Id: I897b9ed35561e455dc6b89c3bacec74b54777903
Signed-off-by: Sorin Sbarnea <ssbarnea@redhat.com>
JJB 2.0 update does not allow updating only views
or jobs which would be useful, which is implemented with
delete and delete-all. This patch add's thoses flags
for the update option.
Change-Id: I3712c8552ea83a4c29c05461894c4464aa41230f
Signed-off-by: Anil Belur <askb23@gmail.com>
Ensure options.path passed to parser is a list.
Otherwise users calling JJB using stdin or another file object will see
a warning about using a deprecated method.
Change-Id: Ibdef0a691a1ca2edc020b8c857e4d77618a000a5
Getting plugin info requires Administrator permissions. It seems the
test command allows providing the plugin_info details via a yaml
configuration file. This patch allows the same command to be passed to
the update command to allow us to configure the plugin versions manually
and not require administrator permissions.
Additionally this patch adds a new command called get-plugins-info
which can create the plugins_info.yaml file to pass to the
plugin_info_path.
See:
http://lists.openstack.org/pipermail/openstack-infra/2016-June/004388.html
Change-Id: I5a34979407d863a84f34afbf8f565081ec52190a
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
Allows user to specify which section from ini config
should be used for loading Jenkins server details.
Change-Id: I57633662e3ca2dba48af8db9e08fb8d5f8bc3bfa
Before, `jenkins-job-builder test -o outdir` would write
outdir/job_name
With the new option `--config-xml`, it writes
outdir/job_name/config.xml
to match the layout that Jenkins uses internally. This is more useful
for diffing against existing Jenkins installations or uploading job
definitions by other means.
Also, it seems better to give the output files an .xml extension.
Change-Id: I78fc7aff531925ee071b8e19f40f766b4af0b5ad
- Adds the ability for JJB to work with views
- Views can be created, updated, and deleted.
- New modules for List view and Build Pipeline view are added
- New tests for testing the deletion of views
Example View configuration:
- view:
name: MyView
view-type: list
Change-Id: Idb29a4407bcc14593e10a4d951036cb04e8e6c27
Co-Authored-By: Brandon Leonard <brandon.leonard@rackspace.com>
Co-Authored-By: Joao Vale <jpvale@gmail.com>
Co-Authored-By: Lucas Dutra Nunes <ldnunes@ossystems.com.br>
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
Since both of these classes essentially build on basic functionality
provided by jenkins.Jenkins, merge them into a single class. Name this
class "JenkinsManager" to avoid confusion with jenkins.Jenkins.
Also moves some tests for update timeout configuration into the
tests.cmd.test_config module from tests.cmd.subcommand.test_update
module, replacing those tests with skipped stubs as reminders to
figure out a better way later.
Change-Id: I13c17bc90e29e702e5e02992e93cf3cdc689731d
Re-add configuration of the logging for CLI, which was accidentally
removed in refactoring.
Remove local instance of logger variable as also defined at the module
level.
Change-Id: Idea548d5a4ed52fd7f9ccb13f9b4e898cd9cc616
Separate XML generation from Yaml parsing/interpreting. The goal here
is to allow different sources to provide data for XML generation,
including external API users writing job definitions in pure Python or
JJB developers who would like to work on alternative Yaml parsing code
since the current YamlParser has very likely reached the limits of
what can be reasonably done with one giant expandYaml procedure.
Change-Id: I9da848acac7e944c0e07286b7399b2e1956a58a5
Create the ModuleRegistry anywhere other than inside the YamlParser
class. This will make it slightly easier to factor a XmlGenerator out
of YamlParser, but I also want to work toward eliminating the circular
references between YamlParser and ModuleRegistry which have been
making it difficult to understand overall program flow.
This commit also replaces all YamlParser instances being passed to
Jenkins job config generating functions with a ModuleRegistry. Mostly
it seems like the parser was only needed to call the ModuleRegistry's
'dispatch' method which to be honest I don't fully understand. This is
where the circular references mentioned in previously come in...it
seems like the "dispatch" function needs access to the (mostly) raw
data contained by the parser, so it took that as a parameter.
The need for the YamlParser's job data can be satisfied by assigning
it to a property on the ModuleRegistry object before Yaml expansion or
XML generation begins; by doing this, we allow the ModuleRegistry to
avoid referencing the parser.
Change-Id: I4b571299b81e708540392ad963163fe092acf1d9
Fixes a regression where using --delete-old will delete jobs that are
not updated for example in the case of caching and jenkins-jobs
determines a job does not need to be updated.
jobs returns a list of jobs that were updated via jenkins-jobs while
xml_jobs returns a list of all jobs generated jobs regardless of if they
were updated or not.
Change-Id: I8f9d0b609d1a7cc0a526d479b4f0a2448c25aba3
Fixes bug introduced by change
Id465078da2a7016fd59407516fe124602abd016c causing all jobs to be
unintentionally deleted on update.
The builder.delete_old_managed()'s keep parameter was expected a
string list of job names but we were mistakenly passing in a list
of XmlJob objects.
Change-Id: I4051925337691d4b59796d21f56cacf4152de1d4
Co-Authored-By: Kien Ha <kienha9922@gmail.com>
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
Sharing code between these two classes will make it easier to work on
extracting an XmlGenerator class from the YamlParser.
Change-Id: If93227d0b4fca4d7bb80de2341d1e83f1e3ddc79
This is the final piece of YamlParser in jenkins_jobs.builder, so also
remove the imports and make trivial fixes to test methods to prevent
borked test cases.
Change-Id: Ida543fa14c70447ae506092ec8afd5cb6caaa629
Make sure that flush cache is correct set first by config file, and
additionally that the CLI overrides any defaults.
In the V2 API work, when moving code around the flush cache option was
accidentally removed from being set by the command line or config
files. This ensures it follows the same behaviour as other similar
options.
Change-Id: I9fb1e234e5ed081ada64855389a87d2f7555469e
This commit sees JJBConfig start to take the form it ought to have,
namely using attributes to represent different logical sections of
configuration that target specific subsystems of JJB.
It also moves ConfigParser data retrieval from
jenkins_jobs.modules.helpers.get_value_from_yaml_or_config_file() to
jenkins_jobs.config.JJBConfig.get_module_config()
TODO: Add JJBConfig tests to validate the behavior of JJBConfig in
specific circumstances.
Change-Id: I053d165559f5325a2f40b239117a86e6d0f3ef37
* Move behavior of jenkins_jobs.cmd.execute() into
jenkins_jobs.cli.entry.JJBConfig.execute()
* Delete jenkins_jobs/cmd.py
* Fix up unit tests to deal with rearranged code.
Change-Id: Ia0d3a062de16c4be10863372c753f4ba8480e620
Ensure that argparser object only makes use of the JJBConfig object
instead of leaking a command line parser into a config object for API
usage.
Take care to use 'None' as default for store_true/store_false options
in order to ensure that the config file defaults are only overridden
when CLI options are explicitly set.
Change-Id: I4066ad750f9893759c2e9bdfde425fafacc7e672
* Create jenkins_jobs.config module with JJBConfig class.
* Move DEFAULT_CONF from jenkins_jobs.cmd into jenkins_jobs.config
* Move configuration initialization into JJBConfig
* Create method, "do_magical_things" to handle arbitration between
config file and arguments as well as setting default values for
config file settings if it doesn't contain the expected keys.
* Move JenkinsJobs.create_parser into its own module,
jenkins_jobs.cli.parser, it can be used to provide default settings
in the JJBConfig class when an argparse namespace object is not
provided; this is primarily necessary because most of the original
configuration initialization code relies on this being a namespace
object (simple descendant of the object class).
At this point JJBConfig isn't much more than an object-oriented
version of the way configuration handling happened previously. Its
current form, however, is more amenable to the ultimate goal of the
2.0 refactorings--namely, being able to pass a single config object
around rather than breaking it up into apparently arbitrary settings
necessary to instantiate the Builder class and its delegate objects.
Change-Id: Ic0147e1dccbe620aaaba039a434e7cea6c670054
This commit intentionally introduces a number of important API
breakages. Specifically, the jenkins_jobs.cmd module has been pared
down to some of its most difficult-to-refactor elements.
* Create jenkins_jobs.cli.entry.JenkinsJobs class to organize command
line parsing and execution.
* Remove references to ConfigParser object in test code, hidden as an
implementation detail of JenkinsJobs command line parsing. This will
be necessary in the next stage of JJB 2.0 code which will be to
create a JJBConfig object that handles logic and presentation of
configuration from various sources--defaults, command line
arguments, configuration file, and maybe environment variables in
the future.
* Remove references to Namespace object produced by argparse module.
Required rewrite of multipath & recursive path tests with a new
MatchesDir testtools Matcher class that validates the expected
output for a run of JJB against a given set of yamldirs with the
specified command line arguments.
* Use stevedore to dynamically load subcommand parsers.
* Move configuration loading/testing to its own test file. Also fix
the global vs home directory JJB config file test.
Change-Id: If62280418ba7319c313033ab387af4284237747e