48 Commits

Author SHA1 Message Date
Vsevolod Fedorov
60e8395c62 Add source location and context to error messages
Change-Id: I2e955c01b71a195bb6ff8ba2bb6f3a64cb3e1f58
2023-04-04 13:35:42 +03:00
Zuul
f3d42c2fbb Merge "Rewrite YAML parser" 2023-03-02 08:49:49 +00:00
Thomas Bechtold
f0b52ec41a Fix --enabled-only option when updating jobs
Commit 5e61fbaf00c97 added a new --enabled-only option but that
doesn't work correctly because the xml_jobs_filtered variable isn't
filled with all the jobs in the end. So some jobs are not updated even
if the disabled status was not found. That's fixed now.

Change-Id: I915116cf875e9e20c10ca43175c924b9e4307cca
2023-03-01 12:50:47 +01:00
Vsevolod Fedorov
af9e03ec08 Rewrite YAML parser
Rewrite YAML parser, YAML objects and parameters expansion logic to
enable better control over expansion logic.
Broken backward compatilibity:
* More agressive parameter expansion. This may lead to parameters
  expanded in places where they were not expanded before.
* Top-level elements, which is not known to parser (such as 'job',
  'view', 'project' etc), are now lead to parse failures.
  Prepend them with underscore to be ignored by parser.
* Files included using '!include-raw:' elements and having formatting in
  it's path ('lazy-loaded' in previous implementation) are now expanded
  too.
  Use '!include-raw-escape:' for them instead.
  See changes in these tests for examples:
    tests/yamlparser/job_fixtures/lazy-load-jobs-multi001.yaml
    tests/yamlparser/job_fixtures/lazy-load-jobs-multi002.yaml
    tests/yamlparser/job_fixtures/lazy-load-jobs001.yaml
* Parameters with template value using itself were substituted as is.
  For example: "timer: '{timer}'" was expanded to "{timer}". Now it
  leads to recursive parameter error.
  See changes in this test for example:
    tests/yamlparser/job_fixtures/parameter_name_reuse_default.*
    ->
    tests/yamlparser/error_fixtures/parameter_name_reuse_default.*
* When job group includes a job which was never declared, it was just
  ignored. Now it fails: job is missing.
  See changes in this test for example:
    tests/yamlparser/job_fixtures/job_group_includes_missing_job.*
    ->
    tests/yamlparser/error_fixtures/job_group_includes_missing_job.*

Change-Id: Ief4e515f065a1b9e0f74fe06d7e94fa77d69f273
2023-02-28 20:16:57 +03:00
Thomas Bechtold
5e61fbaf00 Add config option --enabled-only
That option can be used when calling the update subcommand. If the
option is set, only jobs which are enabled will be updated.
That is useful of there are a lot of disabled jobs which are not
required.

Change-Id: Ie2f268e2356a3b7e0a334a331696ca81e09790aa
2023-02-22 09:11:00 +01:00
wangdonghui
f6556f5636 ListSubCommand: return job fullname
Change-Id: Id1e53d05383efad191ad254ff0f17e3c573aeb65
2022-04-25 19:42:05 +08:00
Thanh Ha
5a98ebb8bb
Fix flake8 BLK100 and BLK999 issues
Resolves flake8 BLK related issues.

"BLK100 Black would make changes."
"BLK999 Unexpected exception: ..."

Change-Id: Ic6d4265e3d02b2f2a15ab851eaed98288c62730c
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
2020-08-31 10:54:19 -04:00
Daniel Bengtsson
0b4ed38ca6 Fix the yaml load warning.
In the new version of PyYAML the API changed to be more explicit. Now
the default value for the Loader is None, see:

https://github.com/yaml/pyyaml/blob/5.1/lib3/yaml/__init__.py#L103

The load is unsafe. It's better to use safe_load function.

Change-Id: Ia1cd16f2ff970ca220a266c99b6554dd4254ebd9
2019-12-10 09:52:40 +01:00
Thanh Ha
4d90c187a9 Auto-generated output from python-black
Please review the following patch containing the code changes in
the repo. This patch is a transition patch and is the auto-generated
output of the python-black tool.

Change-Id: I2d2de71da8a105fb62b561899ae78441ddab4032
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
2019-09-09 19:23:24 +01:00
Thanh Ha
e07ad13ad0
Print config-xml warning once
Instead of printing the deprecation warning against every generated
config-xml file, print it only once at the beginning of the test
command call to not spam the console.

Change-Id: Ifdcdb1373d44ba6232ea7a5359662cc760353774
Signed-off-by: Thanh Ha <zxiiro@linux.com>
2018-11-15 16:28:40 +08:00
Thanh Ha
18a321d510
Fix options.update is not 'all' if no config
If the user does not have 'update' in their configuration
ensure we are properly defaulting to 'all' and not None.

Change-Id: I65ddc49066119db0947801203ac88cb69153d1b0
Signed-off-by: Thanh Ha <zxiiro@linux.com>
2018-09-19 09:46:14 -04:00
Anil Belur
9399b1f986
Add config options --jobs-only and --views-only
Add options --jobs-only and --views-only to job_builder
section in config file.

By default JJB updates both jobs and views. Some cases
non-admin users are not allowed to update views on Jenkins
which requires explicity using -j flag.

Allow users to set a config option 'update=jobs|views|all'
in the 'job_builder' section implicitly use --jobs-only or
--views-only and control these flags from the config file.

Note: CLI options takes precedence over the config file.

Change-Id: I2a94e5a2d671ccbfc505de2f19b578ecfef9e9d7
Co-Authored-By: Thanh Ha <zxiiro@linux.com>
Signed-off-by: Anil Belur <askb23@gmail.com>
Signed-off-by: Thanh Ha <zxiiro@linux.com>
Signed-off-by: Anil Belur <askb23@gmail.com>
2018-09-19 02:24:05 +05:30
Sorin Sbarnea
92f28a591e
Allow jjb to be called as a module
Enable us to do `python -m jenkins_jobs ...`
which avoids the need to install executable in
user path.

Tests cli execution on each supported python version.

Uses default encoding workaround only on
python2, where is needed.

Change-Id: I4cd79fd51a8309d532e0e76723ecfbbda3e1ca6f
Signed-off-by: Sorin Sbarnea <ssbarnea@redhat.com>
2018-06-19 23:25:43 +01:00
Sorin Sbarnea
4f7b6ee568
upgrade hacking module
- upgrades hacking to current version
- sorts new linting issues
- sorts bug with sys.reload on py3

Change-Id: I4a18abc93116667a2733e8aec619ac59ea73d630
Signed-off-by: Sorin Sbarnea <ssbarnea@redhat.com>
2018-06-19 18:04:37 +01:00
Sorin Sbarnea
144ace119b
Improve unicode support
Printed job urls with unicode are now clickable

Avoids unicode decode errors in logging due on
systems, likely python27 systems where
default encoding is ascii.

Change-Id: I17041de05a4388c7b95a9ca04477043424458c67
2018-06-17 15:53:02 +01:00
Truc Nguyen
c051dd8502
Fixes delete-all command when no option is provided
Change-Id: Ibc06209c2ec19045e12a145083a64a179d57d731
Implements: fix for delete-all command without option
Closes-Bug: #2001207
2018-05-31 19:07:12 +01:00
Zuul
60602ea938 Merge "Add 'existing-only' argument for update subcommand" 2018-04-12 19:59:13 +00:00
Malcolm Brooks
fc1f6fde54 Add 'existing-only' argument for update subcommand
Additional argument for the update subcommand to allow user to update
only jobs currently installed on the Jenkins server and managed by
this tool.

Change-Id: I4b0e997df01057f89896b8caa27dc00b5ca077df
2018-04-04 19:01:50 +01:00
Sorin Sbarnea
d47634fc3f Allows params to be specified via environment variables
Adds ability to load common params from environment variables:
JJB_CONF
JJB_LOG_LEVEL
JJB_SECTION
JJB_USER
JJB_PASSWORD

Change-Id: Ida786f2e60e4b3e54d739436d0970917a5f93518
2018-03-27 12:53:02 +01:00
Marc Abramowitz
8d4671e3a7
Add a "list" subcommand
Add a list subcommand that allows listing of all the jobs defined in
yaml configuration, or on the Jenkins master, with the option of taking
glob parameters to allow matching some job names.

Also supports the recursive and exclude path control options.

  jenkins-jobs list 'wikimedia-fundraising*'

  jenkins-jobs list -r -p configs/ 'wikimedia-fundraising*'

Co-Authored-By: Darragh Bailey <dbailey@hpe.com>
Co-Authored-By: Sorin Sbarnea <ssbarnea@redhat.com>
Change-Id: I897b9ed35561e455dc6b89c3bacec74b54777903
Signed-off-by: Sorin Sbarnea <ssbarnea@redhat.com>
2018-03-09 10:58:21 -05:00
Sorin Sbarnea
c6ff48ce4a Added main entry point
This allows using external debuggers.

Change-Id: I8d29534b0ef22c3ddb0e70c7fafe94f49c0d7f4e
Signed-off-by: Sorin Sbarnea <ssbarnea@redhat.com>
2018-02-28 16:17:21 +00:00
Anil Belur
ad61af1670
Add views and jobs only options for update
JJB 2.0 update does not allow updating only views
or jobs which would be useful, which is implemented with
delete and delete-all. This patch add's thoses flags
for the update option.

Change-Id: I3712c8552ea83a4c29c05461894c4464aa41230f
Signed-off-by: Anil Belur <askb23@gmail.com>
2018-01-25 17:18:12 +10:00
Darragh Bailey
6204001bc2 Prevent deprecation warning using test with stdin
Ensure options.path passed to parser is a list.

Otherwise users calling JJB using stdin or another file object will see
a warning about using a deprecated method.

Change-Id: Ibdef0a691a1ca2edc020b8c857e4d77618a000a5
2017-12-15 14:27:08 +00:00
Thanh Ha
2974f1e636
Allow update to provide plugin_info via yaml
Getting plugin info requires Administrator permissions. It seems the
test command allows providing the plugin_info details via a yaml
configuration file. This patch allows the same command to be passed to
the update command to allow us to configure the plugin versions manually
and not require administrator permissions.

Additionally this patch adds a new command called get-plugins-info
which can create the plugins_info.yaml file to pass to the
plugin_info_path.

See:
http://lists.openstack.org/pipermail/openstack-infra/2016-June/004388.html

Change-Id: I5a34979407d863a84f34afbf8f565081ec52190a
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
2017-09-19 18:17:15 -04:00
Jenkins
3985a338c4 Merge "Add option to make test output file layout match Jenkins" 2017-06-08 05:24:33 +00:00
Sorin Sbarnea
5d3aa0dbb5 Add support for using multiple servers
Allows user to specify which section from ini config
should be used for loading Jenkins server details.

Change-Id: I57633662e3ca2dba48af8db9e08fb8d5f8bc3bfa
2017-05-30 16:12:51 +01:00
Peter Eisentraut
919052714c Add option to make test output file layout match Jenkins
Before, `jenkins-job-builder test -o outdir` would write

    outdir/job_name

With the new option `--config-xml`, it writes

    outdir/job_name/config.xml

to match the layout that Jenkins uses internally.  This is more useful
for diffing against existing Jenkins installations or uploading job
definitions by other means.

Also, it seems better to give the output files an .xml extension.

Change-Id: I78fc7aff531925ee071b8e19f40f766b4af0b5ad
2017-05-27 12:22:06 +01:00
Sorin Sbarnea
8a7d77c180 Removed some deprecation warnings and enabled py35 on tox.
Change-Id: Icc5b28c4636b542a10502b7b1d2d2dc2028cc166
Signed-off-by: Sorin Sbarnea <ssbarnea@redhat.com>
2017-03-12 19:43:33 +00:00
Thanh Ha
1deb3aff4c
Add view management functionality
- Adds the ability for JJB to work with views
- Views can be created, updated, and deleted.
- New modules for List view and Build Pipeline view are added
- New tests for testing the deletion of views

Example View configuration:

    - view:
        name: MyView
        view-type: list

Change-Id: Idb29a4407bcc14593e10a4d951036cb04e8e6c27
Co-Authored-By: Brandon Leonard <brandon.leonard@rackspace.com>
Co-Authored-By: Joao Vale <jpvale@gmail.com>
Co-Authored-By: Lucas Dutra Nunes <ldnunes@ossystems.com.br>
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
2016-10-14 09:14:21 -04:00
Wayne Warren
e93f4a5f5a
Convert multiline strings to concatenated single-quote.
Change-Id: Ic0c1be9ca037b67695136583837702d3ec4d42a8
2016-09-14 15:39:36 -04:00
Thanh Ha
e47098d89d Simplify delete by removing unnecessary loop
Change-Id: Ic0d2ee0863c012bba22599e7be94718033c4aa97
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
2016-09-14 14:34:34 +01:00
Wayne Warren
e75062d103 Merge builder.Builder and builder.Jenkins
Since both of these classes essentially build on basic functionality
provided by jenkins.Jenkins, merge them into a single class. Name this
class "JenkinsManager" to avoid confusion with jenkins.Jenkins.

Also moves some tests for update timeout configuration into the
tests.cmd.test_config module from tests.cmd.subcommand.test_update
module, replacing those tests with skipped stubs as reminders to
figure out a better way later.

Change-Id: I13c17bc90e29e702e5e02992e93cf3cdc689731d
2016-09-02 08:58:09 -07:00
Wayne Warren
2236a00c53 Rename Builder.delete_job to Builder.delete_jobs.
Change-Id: If57a98a00e9577fdf7a3d7585905a2102d8c61cd
2016-08-20 11:28:00 -07:00
Darragh Bailey
72acdc2149 Ensure logging enabled for CLI
Re-add configuration of the logging for CLI, which was accidentally
removed in refactoring.

Remove local instance of logger variable as also defined at the module
level.

Change-Id: Idea548d5a4ed52fd7f9ccb13f9b4e898cd9cc616
2016-08-19 17:16:04 +01:00
Wayne Warren
aaae83c623
Factor XmlJobGenerator out of YamlParser.
Separate XML generation from Yaml parsing/interpreting. The goal here
is to allow different sources to provide data for XML generation,
including external API users writing job definitions in pure Python or
JJB developers who would like to work on alternative Yaml parsing code
since the current YamlParser has very likely reached the limits of
what can be reasonably done with one giant expandYaml procedure.

Change-Id: I9da848acac7e944c0e07286b7399b2e1956a58a5
2016-08-18 22:42:24 -04:00
Wayne Warren
ae1fb60f16
Disentangle YamlParser and ModuleRegistry classes
Create the ModuleRegistry anywhere other than inside the YamlParser
class. This will make it slightly easier to factor a XmlGenerator out
of YamlParser, but I also want to work toward eliminating the circular
references between YamlParser and ModuleRegistry which have been
making it difficult to understand overall program flow.

This commit also replaces all YamlParser instances being passed to
Jenkins job config generating functions with a ModuleRegistry. Mostly
it seems like the parser was only needed to call the ModuleRegistry's
'dispatch' method which to be honest I don't fully understand. This is
where the circular references mentioned in previously come in...it
seems like the "dispatch" function needs access to the (mostly) raw
data contained by the parser, so it took that as a parameter.

The need for the YamlParser's job data can be satisfied by assigning
it to a property on the ModuleRegistry object before Yaml expansion or
XML generation begins; by doing this, we allow the ModuleRegistry to
avoid referencing the parser.

Change-Id: I4b571299b81e708540392ad963163fe092acf1d9
2016-08-18 22:42:24 -04:00
Timothy R. Chavez
d2d7c3f3f9
Use xml_jobs not jobs
Fixes a regression where using --delete-old will delete jobs that are
not updated for example in the case of caching and jenkins-jobs
determines a job does not need to be updated.

jobs returns a list of jobs that were updated via jenkins-jobs while
xml_jobs returns a list of all jobs generated jobs regardless of if they
were updated or not.

Change-Id: I8f9d0b609d1a7cc0a526d479b4f0a2448c25aba3
2016-08-15 23:12:27 -04:00
Thanh Ha
80aa526616
Fix bug causing all jobs to be deleted
Fixes bug introduced by change
Id465078da2a7016fd59407516fe124602abd016c causing all jobs to be
unintentionally deleted on update.

The builder.delete_old_managed()'s keep parameter was expected a
string list of job names but we were mistakenly passing in a list
of XmlJob objects.

Change-Id: I4051925337691d4b59796d21f56cacf4152de1d4
Co-Authored-By: Kien Ha <kienha9922@gmail.com>
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
2016-08-07 00:11:19 -04:00
Wayne Warren
4350690b61 TestSubCommand inherit from UpdateSubCommand.
Sharing code between these two classes will make it easier to work on
extracting an XmlGenerator class from the YamlParser.

Change-Id: If93227d0b4fca4d7bb80de2341d1e83f1e3ddc79
2016-08-05 08:56:38 -07:00
Wayne Warren
99fd084099 Move subcommand logic into subcommand classes.
Also, add copyright notices to subcommand files and fix mock import
paths in tests.

Change-Id: Ia8a96d5d5c460387968437d86cf7144ef382b20c
2016-08-05 08:32:51 -07:00
Wayne Warren
7373201882 Move 'delete' yaml parsing to JenkinsJobs.execute.
This is the final piece of YamlParser in jenkins_jobs.builder, so also
remove the imports and make trivial fixes to test methods to prevent
borked test cases.

Change-Id: Ida543fa14c70447ae506092ec8afd5cb6caaa629
2016-08-05 08:32:46 -07:00
Wayne Warren
3ce6618381 Move 'update' yaml parsing to JenkinsJobs.execute.
1 Remove YamlParser from Builder.
 2 ...
 3 Rule the world.

Change-Id: Id465078da2a7016fd59407516fe124602abd016c
2016-08-05 06:40:12 -07:00
Darragh Bailey
652806a751 Ensure flush cache option obeyed
Make sure that flush cache is correct set first by config file, and
additionally that the CLI overrides any defaults.

In the V2 API work, when moving code around the flush cache option was
accidentally removed from being set by the command line or config
files. This ensures it follows the same behaviour as other similar
options.

Change-Id: I9fb1e234e5ed081ada64855389a87d2f7555469e
2016-07-22 17:55:22 +01:00
Wayne Warren
4f04de1f9a Use JJBConfig in YamlParser.
This commit sees JJBConfig start to take the form it ought to have,
namely using attributes to represent different logical sections of
configuration that target specific subsystems of JJB.

It also moves ConfigParser data retrieval from
jenkins_jobs.modules.helpers.get_value_from_yaml_or_config_file() to
jenkins_jobs.config.JJBConfig.get_module_config()

TODO: Add JJBConfig tests to validate the behavior of JJBConfig in
specific circumstances.

Change-Id: I053d165559f5325a2f40b239117a86e6d0f3ef37
2016-07-22 17:33:25 +01:00
Wayne Warren
b6e9080a89 Finish removing code from jenkins_jobs.cmd
* Move behavior of jenkins_jobs.cmd.execute() into
  jenkins_jobs.cli.entry.JJBConfig.execute()
* Delete jenkins_jobs/cmd.py
* Fix up unit tests to deal with rearranged code.

Change-Id: Ia0d3a062de16c4be10863372c753f4ba8480e620
2016-07-22 17:15:36 +01:00
Darragh Bailey
70334fbf4b Untangle argparser and config objects
Ensure that argparser object only makes use of the JJBConfig object
instead of leaking a command line parser into a config object for API
usage.

Take care to use 'None' as default for store_true/store_false options
in order to ensure that the config file defaults are only overridden
when CLI options are explicitly set.

Change-Id: I4066ad750f9893759c2e9bdfde425fafacc7e672
2016-07-22 15:45:48 +01:00
Wayne Warren
2cf11f014c Use JJBConfig for arg/config initialization.
* Create jenkins_jobs.config module with JJBConfig class.
* Move DEFAULT_CONF from jenkins_jobs.cmd into jenkins_jobs.config
* Move configuration initialization into JJBConfig
* Create method, "do_magical_things" to handle arbitration between
  config file and arguments as well as setting default values for
  config file settings if it doesn't contain the expected keys.
* Move JenkinsJobs.create_parser into its own module,
  jenkins_jobs.cli.parser, it can be used to provide default settings
  in the JJBConfig class when an argparse namespace object is not
  provided; this is primarily necessary because most of the original
  configuration initialization code relies on this being a namespace
  object (simple descendant of the object class).

At this point JJBConfig isn't much more than an object-oriented
version of the way configuration handling happened previously. Its
current form, however, is more amenable to the ultimate goal of the
2.0 refactorings--namely, being able to pass a single config object
around rather than breaking it up into apparently arbitrary settings
necessary to instantiate the Builder class and its delegate objects.

Change-Id: Ic0147e1dccbe620aaaba039a434e7cea6c670054
2016-07-19 18:57:34 +01:00
Wayne
fc73cedb45 Introduce modular implementation of subcommands.
This commit intentionally introduces a number of important API
breakages. Specifically, the jenkins_jobs.cmd module has been pared
down to some of its most difficult-to-refactor elements.

* Create jenkins_jobs.cli.entry.JenkinsJobs class to organize command
  line parsing and execution.
* Remove references to ConfigParser object in test code, hidden as an
  implementation detail of JenkinsJobs command line parsing. This will
  be necessary in the next stage of JJB 2.0 code which will be to
  create a JJBConfig object that handles logic and presentation of
  configuration from various sources--defaults, command line
  arguments, configuration file, and maybe environment variables in
  the future.
* Remove references to Namespace object produced by argparse module.
  Required rewrite of multipath & recursive path tests with a new
  MatchesDir testtools Matcher class that validates the expected
  output for a run of JJB against a given set of yamldirs with the
  specified command line arguments.
* Use stevedore to dynamically load subcommand parsers.
* Move configuration loading/testing to its own test file. Also fix
  the global vs home directory JJB config file test.

Change-Id: If62280418ba7319c313033ab387af4284237747e
2016-07-08 09:55:44 -07:00