152 Commits

Author SHA1 Message Date
Wayne Warren
c569dc02d3 Move load_files method to YamlParser from Builder.
This is the first step in removing jenkins_modules.parser.YamlParser
references from jenkins_jobs.builder.Builder entirely, necessary
because load_files is actually a YamlParser behavior, not a Builder
behavior.

Change-Id: I1cad99b4cdb44af25ba398837f7f328cfcbd5bbb
2016-07-22 17:33:53 +01:00
Wayne Warren
4f04de1f9a Use JJBConfig in YamlParser.
This commit sees JJBConfig start to take the form it ought to have,
namely using attributes to represent different logical sections of
configuration that target specific subsystems of JJB.

It also moves ConfigParser data retrieval from
jenkins_jobs.modules.helpers.get_value_from_yaml_or_config_file() to
jenkins_jobs.config.JJBConfig.get_module_config()

TODO: Add JJBConfig tests to validate the behavior of JJBConfig in
specific circumstances.

Change-Id: I053d165559f5325a2f40b239117a86e6d0f3ef37
2016-07-22 17:33:25 +01:00
Sebastian Schuberth
df1ab0f01a Catch OSError when renaming and try to remove the destination file first
Otherwise the rename fails on Windows if the destination file already
exists.

Change-Id: Id15cfaa5b9278d7e932a7975ab6e5a878b31a642
2016-06-09 11:49:30 +02:00
Darragh Bailey
7d13236760 Python 3.x tempfile requires unicode
Explicitly encode the output of yaml dump, and allow tempfile to handle
the unicode output as it works correctly for python 2.7 and 3.x+

Change-Id: I919b2565a7f3857586ee6cc09b787160472e9deb
2016-05-25 19:15:22 +01:00
Darragh Bailey
85ee46297c Prevent corruption of cache by using atomic update
Ensure that cache is written to a temporary file and use an atomic OS
file rename operation to ensure that the cache is either the old
contents or new contents and cannot be corrupted by an exception
occurring during writing or should the process be killed.

Change-Id: I69947cc6d80fdc80ee7addde3a2ff87cd9c3297b
2016-05-20 14:43:25 -07:00
Jenkins
a725ca89af Merge "Wrap stdout object only once with wrap_stream" 2016-04-21 23:15:11 +00:00
Clark Boylan
5407302fdc Revert "Allow using lockfile per jenkins master"
This reverts commit d1df3359b3c9fd968e8e35c51dc36b0b90ffa216.

The lockfile management code attempts to do IO on a closed file and
throws errors.

Change-Id: Ied58185758fd7c8c822624e169216441341a75d0
2016-03-16 10:26:18 -07:00
Jenkins
0c8ce8ce8f Merge "Allow using lockfile per jenkins master" 2016-03-16 00:11:17 +00:00
Yolanda Robla
d1df3359b3 Allow using lockfile per jenkins master
When a jjb run is thrown when another jjb is already running,
it can cause corruption of cache. Start using a lockfile to
ensure this won't be happening and run securely on automated
systems.

Change-Id: I3ac37e738b3bb87c04a47afb8adb3e25f8fb4ea8
2016-03-14 17:28:39 +01:00
Joakim Löfgren
aa3bf5cc19 Fix issue with non-ascii characters in get_job_md5
The xml string needs to be encoded to utf-8
before passing it to the hash function.

Change-Id: I4b6ca7ef459b48a2ba56f788fe37307ee381ced4
Story: 2000488
2016-02-23 09:10:25 +01:00
Thanh Ha
bd1c93199a
Fix delete-all command to include all job types
getProjects() only includes simple job types like Freestyle jobs but
ignores more complex job types like Maven and Matrix jobs. This API is
also marked as deprecated in Jenkins so we probably shouldn't be using
it anyway.

Instead switch to using getAllItems() which pulls down all job types. It
is also possible to pass a parameter such as AbstractProjects or more
specific to specify exactly which job types we want to remove. In this
case though we want all of them so no need to pass any parameters.

See:
http://javadoc.jenkins-ci.org/jenkins/model/Jenkins.html

Change-Id: Ia4e37bc1d4bde70cf8f83783edd6b861eed6c2e3
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
2016-02-03 19:25:47 -05:00
Darragh Bailey
e1f5dec356 Wrap stdout object only once with wrap_stream
Make sure to perform wrap external to loop, as otherwise this will
recursively wrap the output object and potentially result in getattr
lookups exceeding the recursion limit in python.

Change-Id: Ie9318a3d23a721e62c16e7b05f52d5b9bec1bb82
2016-01-11 18:35:20 +00:00
Jenkins
71449dbed9 Merge "Distinguish kept jobs and unmanaged jobs when skipping removal" 2016-01-11 16:48:24 +00:00
Darragh Bailey
8805a34c0f Reorder imports to match hacking guidelines
Ensure that the imports follow the standard OpenStack hacking
guidelines.

Change-Id: Iaa4326aef118ddfd807dd006934f1d9ca80a1cfa
2015-12-23 15:23:42 -08:00
David Caro
333fdc65a1
Add parallelization options
* Only update_jobs uses the parallel features right now
* --workers N
    - If set to 0, it will use parallel execution and
      use the number of cores in the machine as thread count
    - Any other value enables the parallel extensions and sets the
      number of concurrent threads to that value
    - Will use 1 by default

Add some tests to make sure the parallel execution helpers work as
expected.

Change-Id: Ib0abd34ea7525f75fff4ff480287a6e589deba90
Signed-off-by: David Caro <dcaroest@redhat.com>
2015-12-01 10:05:04 -05:00
Jenkins
511c3ad577 Merge "builder.py: fix potential race condition" 2015-11-14 06:46:06 +00:00
Khai Do
899bc6c946 fix for python 3, decode job output
Change Ie952617a34c0719e forced utf-8 format but forgot to decode the
xml when passing to update_job. This broke jjb for python 3:

 File "/Users/khaido/PycharmProjects/jenkins-job-builder
     /jenkins_jobs/builder.py", line
 134, in update_job
    self.jenkins.create_job(job_name, xml)

 File "/Users/khaido/PycharmProjects/jenkins-job-builder/.tox
      /py34/lib/python3.4/site-packages/jenkins/__init__.py",
 line 852, in create_job
      config_xml.encode('utf-8'), DEFAULT_HEADERS))
  AttributeError: 'function' object has no attribute 'encode'

Change-Id: Iceda46214bf4709ccd8141ef654cf3ec81e8af06
2015-10-13 00:26:20 -07:00
Jenkins
4672157c05 Merge "Fix update after deleting all jobs" 2015-10-12 17:20:02 +00:00
Khai Do
247cb35fe8 Fix update after deleting all jobs
Change Id7f7dfb567997e0 introduced a bug where JJB thought that the jobs
were still on the Jenkins server after using the deleting-all command.
Subsequent updates would not re-create the jobs.  This change fixes that
problem by clearing out the JJB cache after all jobs have been removed.

Change-Id: I37e65a78f4ba8b05d76b74ff9841d45de9b5bfad
2015-10-09 16:04:17 -07:00
Darragh Bailey
b023d7e23f Wrap file objects with codecs to handle unicode
Ensure file objects including stdin/stdout objects are wrapped using
codecs to handle unicode translation to the selected encoding for
input/output.

Add tests to simulate different encodings for input/output and
consequently fix the reading of input from stdin. Include test to
trigger failure to encode a unicode valid character using 'ascii'
encoding.

Change-Id: I9a5c4c96d131873db0000377729b8b0a45aa470d
2015-10-09 10:48:01 -07:00
Jonathan Lebon
641eb75b69 builder.py: fix potential race condition
There is a possible TOCTOU race condition in the event that two
jenkins-jobs instances are running at the same time and try to create
the cache directory at the same time.

story: 2000374
Change-Id: Ie194eb7b52237356e8ab935aed97af577a96605b
2015-10-08 12:49:07 -04:00
Darragh Bailey
647bbdb7fc Distinguish kept jobs and unmanaged jobs when skipping removal
Distinguish between ignoring kept managed jobs and unmanaged jobs when
deleting old jobs from the master.

Change-Id: I41e809a775ec9d07863b7c532c98f2eac4c8e5bf
2015-09-29 18:57:00 +01:00
Khai Do
55e0fdc86c Allow JJB to set connection timeout to jenkins server
Change If84778231b provided an option to set a connection timeout
in the python-jenkins library.  This change allows JJB to override
that timeout value.

Add tests to ensure that the timeout is only set when specified via the
config options.

Change-Id: I3dfe9139469dd0e8549eeedb1833c55ac79ea8b5
2015-09-21 09:36:55 -07:00
Jenkins
b8eb60239f Merge "Improve delete all job performance" 2015-09-18 17:35:02 +00:00
Jenkins
8c46ff8b10 Merge "Replace open() with io.open() and force 'utf-8'" 2015-08-10 18:14:11 +00:00
Khai Do
67de07ceed Improve delete all job performance
delete-all command was deleting one job at a time using the jenkins
doDelete rest endpoint which is very slow.  Change Ia4fbfca970165
allows us to execute a groovy script on the jenkins server to
do the same thing.  The only difference is that running the script
on the server will delete all jobs much faster.

As a data point:
I added all (~5,000) openstack jobs to my jenkins server.
The previous delete_all took many hours to delete all the
jobs while the script method did it in a matter of seconds.

Change-Id: Id7f7dfb567997e042fe0f783e54680482fefdc15
depends-on: Ia4fbfca970165d890d7e076f47ddcde7633afa9b
2015-07-31 14:48:30 -07:00
Darragh Bailey
0e74bddb56 Replace open() with io.open() and force 'utf-8'
Use io.open() to allow reading and writing of files in 'utf-8' format
irrespective of the terminal encoding selected.

Change-Id: Ie952617a34c0719efc59a7729d698beafaa477b0
2015-07-25 14:16:12 -07:00
David Caro
32f402f3e5 Removed unnecessary parameter from delete_old_jobs
Also removed the extra return value from the update_jobs method.

Change-Id: Ide838ec5313f736f7a9b4e696f82beb3d9805d57
Signed-off-by: David Caro <dcaroest@redhat.com>
2015-07-08 14:52:17 +01:00
Jenkins
e460a1e1ae Merge "Move writexml fix for older minidom to same module as caller" 2015-06-19 17:26:58 +00:00
Yolanda Robla
6c6370bbff Query all jobs once
Instead of doing an individual api call each time,
to check if a job exists, do an initial jenkins get_jobs()
call on init, and use that list as a cache, to speed
up job queries.

If a job is not found in cache, we still will be querying
jenkins directly as a fallback.

Change-Id: Ie3eff875f4d832a42c9e38478f62055bc135a7e0
2015-05-12 12:50:10 +02:00
James E. Blair
f74ce34d36 Revert "Added parallelization options"
This reverts commit 7100a7f225e60936f23d24c757c1b14c9b136ec4.

The referenced commit, in addition to what it advertises, also
filters the result of update_jobs to include only changed jobs.
If there are no change jobs, then, if the user specifies
--delete-old, there are no jobs in the list of jobs to keep
which comes from the result of update_jobs.  In that case, all
jobs are deleted.  Or in other words, the logic in this change
is that all jobs not updated are deleted.

Proposing as a revert because this is currently causing serious
problems for continuously deployed JJB instances, and the very
common --delete-old code path should be properly tested with
this change before it lands again.

Change-Id: I98443f0c085e27ed8dfece6409434015ac24b306
2015-05-05 18:30:03 +00:00
Darragh Bailey
571a0efd16 Move writexml fix for older minidom to same module as caller
With the reorganization of code to move the XmlJob class to a separate
file, the patch to modify the minidom method writexml for earlier
instances that incorrectly added whitespace should be relocated to be
alongside.

This ensures the patch exists along with the code that is using
minidom's functionality.

Change-Id: I6ed3267c3c076dc5f8d40f0c97706c33e4018261
2015-05-05 14:32:21 +01:00
David Caro
7100a7f225 Added parallelization options
* Only update_jobs uses the parallel features right now
* --workers N
    - if set to 0, it will use parallel execution and
      use the number of cores in the machine as thread count
    - Any other value enables the paralllel extensions and sets the
      number of concurrent threads to that value
    - Will use 1 by default

Added some tests to make sure the parallel execution helpers work as
expected

Change-Id: Iee8426ff2dc4c5b4ebec4c15bdd21cae5b47eb83
Signed-off-by: David Caro <dcaroest@redhat.com>
2015-04-30 21:33:08 +02:00
Jenkins
f6a8cfede5 Merge "Remove YamlParser from jenkins_jobs.builder" 2015-04-23 17:07:59 +00:00
Jenkins
38ef922a2b Merge "Remove superfluous calls to matches()" 2015-04-23 17:07:10 +00:00
Jenkins
4d57933124 Merge "Log the job name info when running test mode" 2015-04-23 15:20:00 +00:00
Wayne
4a8b93b8c2 Remove YamlParser from jenkins_jobs.builder
The goal of this patch is simply to move some classes out of
jenkins_jobs.builder into more appropriately-named modules. This started with
simply moving YamlParser into jenkins_jobs.parser but led to other moves in
order to avoid cyclic imports since YamlParser uses other classes previously
defined in jenkins_jobs.builder.

That said, this patch doesn't intend to address all of the clutter in
jenkins_jobs.builder, mostly just what is necessary to get started working on
YamlParser independent of other classes in that module.

Change-Id: Ie88bf683e495033eb0b670fe29c256a70282735f
2015-04-22 11:59:07 -07:00
Darragh Bailey
c29066210c Remove superfluous calls to matches()
Remove unnecessary calls to matches() and ensure that delete_job() is
correctly passing the jobs_glob argument to the expandYaml() method call
since it uses a string whereas update_job() is using a sequence of
strings.

Once the bug with how delete_job is calling expandYaml() is fixed, the
subsequent call to matches() can be removed.

Change-Id: Iaf1404e26691384fd1d55ea630457860cc1beff5
2015-04-22 18:27:31 +01:00
Khai Do
9ebba4106d Log info about updated and deleted jobs
Log the number of jobs that were updated and deleted on use of the
update command.

Change-Id: I7d6e955556873749629b8942a18a6811c2d15ee1
2015-04-16 11:07:59 -07:00
Jenkins
64f681a67d Merge "Stop mkdir output on each job write" 2015-04-16 13:38:04 +00:00
David Caro
814ba7575f Added possibility to use non-existent keys
Added a new configuration option under the section "job_builder" named
"allow_empty_variables" that if set to true, will replace non existing variables
in strings with the empty string instead of raising an error.

It's very useful if you  have a shell script, that has optional values, for
example:

 EXTRA_PACKAGES=({extra-packages})
 for package in "${EXTRA_PACKAGES[@]}"; do
    install "$package"
 done

Or modifying the script behavior with flags, with empty as default value:

 WITH_AUTOGEN={with-autogen}
 if [[ $WITH_AUTOGEN ]]; then
    ./autogen.sh
 fi

Then if you have two different jobs that use that script in their builder, you
just have to set the extra parameter or ignore it to change the behavior:

 - builder:
   name: mybuilder
   builders:
      - shell: !include shell-scripts/myscript.sh

 - job-template:
   name: 'mytpl-{name}'
   ...
   builders:
     - mybuilder

 - project:
   name: myproj1
   with-autogen: true
   jobs:
     - 'mytpl-{name}'

 - project:
   name: myproj2
   extra-packages: |
     extrapkg1
     extrapkg2

Change-Id: Iad9f0e522725e6fd6681cd62d3e36f69baf09585
Signed-off-by: David Caro <dcaroest@redhat.com>
2015-03-24 19:20:49 +01:00
Antoine Musso
750b576246 Stop mkdir output on each job write
When JJB write jobs to an output directory, each job would attempt to
mkdir the output directory causing unneccessary system calls:

mkdir("output-parent", 0777)            = -1 EEXIST (File exists)
stat("output-parent", {st_mode=S_IFDIR|0755, ...}) = 0

Create the output directory before the write loop unless it the output
is writable (stdout) or it already exist. That should speed up the test
operation a little bit.

Change-Id: I17495cba51bd85e4a6f86491a992cec6a3024e72
2015-03-24 15:34:00 +01:00
Khai Do
a8f7e4a98f Log the job name info when running test mode
When user runs 'jenkins-jobs test myjobs.yaml' only the xml
gets printed to screen but the xml doesn't contain any info
about the job name.  This logs the job name before showing
the xml snippet for each job.  Helps in debugging jobs when
using templates.

For example:

INFO:jenkins_jobs.builder:Job name:  foojob-py27-unit-master
<?xml version="1.0" encoding="utf-8"?>
<project>
  <actions/>
  <description>&lt;!-- Managed by Jenkins Job Builder --&gt;</description>
  <keepDependencies>false</keepDependencies>
  <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
  <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
  <concurrentBuild>false</concurrentBuild>
  <canRoam>true</canRoam>
  <properties/>
  <scm class="hudson.scm.NullSCM"/>
  <builders>
    <hudson.tasks.Shell>
      <command>git checkout master</command>
    </hudson.tasks.Shell>
  </builders>
  <publishers/>
  <buildWrappers/>
</project>
..
..

Change-Id: Ibce45debd5fd6aa1a400e1cc3762846415ca2a75
2015-03-12 10:19:15 -07:00
Jenkins
298ba7c6b0 Merge "Fix typo in format string" 2015-03-05 17:35:03 +00:00
Jenkins
db9d6f7ed6 Merge "Skip duplicate input files" 2015-03-05 17:34:47 +00:00
Jenkins
c116037dfd Merge "Provide info on number of jobs processed for test, update and delete-all" 2015-03-05 15:51:30 +00:00
James E. Blair
41fe32128f Fix typo in format string
Change-Id: I79ed5033a3673c56973eb6908e5969ba33258aa6
2015-03-05 07:32:46 -08:00
Darragh Bailey
302255ac60 Check external documentation links are valid
Use sphinx's linkcheck builder to validate any external links.

Fix the broken links by converting the example link to a code segment
and correcting the link referencing the cgit mirror for the git clone
instructions.

Change-Id: Ic6236e250b7af5c7c000af1e67fb69dd720ad6d8
2015-02-24 21:36:54 +00:00
Wayne
c1b290ff13 Fix bug in template job yaml expansion.
This bug occurs if one attempts to set a list object on a job template for a
parameter that does not occur in the job name. If the parameter is not in the
template name then there is no reason to count it among the dimensions used to
fanout that template into multiple instances and in fact this would actually
lead to confusing duplicate job error messages.

Change-Id: I80f026d3bbfbac96c6fc7c01c1c916cf85e1bf10
2015-02-22 17:47:06 -08:00
Jenkins
d2f69bc699 Merge "Report source of duplicate entries where possible" 2015-02-20 20:19:11 +00:00