This is the first step in removing jenkins_modules.parser.YamlParser
references from jenkins_jobs.builder.Builder entirely, necessary
because load_files is actually a YamlParser behavior, not a Builder
behavior.
Change-Id: I1cad99b4cdb44af25ba398837f7f328cfcbd5bbb
This commit sees JJBConfig start to take the form it ought to have,
namely using attributes to represent different logical sections of
configuration that target specific subsystems of JJB.
It also moves ConfigParser data retrieval from
jenkins_jobs.modules.helpers.get_value_from_yaml_or_config_file() to
jenkins_jobs.config.JJBConfig.get_module_config()
TODO: Add JJBConfig tests to validate the behavior of JJBConfig in
specific circumstances.
Change-Id: I053d165559f5325a2f40b239117a86e6d0f3ef37
Explicitly encode the output of yaml dump, and allow tempfile to handle
the unicode output as it works correctly for python 2.7 and 3.x+
Change-Id: I919b2565a7f3857586ee6cc09b787160472e9deb
Ensure that cache is written to a temporary file and use an atomic OS
file rename operation to ensure that the cache is either the old
contents or new contents and cannot be corrupted by an exception
occurring during writing or should the process be killed.
Change-Id: I69947cc6d80fdc80ee7addde3a2ff87cd9c3297b
This reverts commit d1df3359b3c9fd968e8e35c51dc36b0b90ffa216.
The lockfile management code attempts to do IO on a closed file and
throws errors.
Change-Id: Ied58185758fd7c8c822624e169216441341a75d0
When a jjb run is thrown when another jjb is already running,
it can cause corruption of cache. Start using a lockfile to
ensure this won't be happening and run securely on automated
systems.
Change-Id: I3ac37e738b3bb87c04a47afb8adb3e25f8fb4ea8
The xml string needs to be encoded to utf-8
before passing it to the hash function.
Change-Id: I4b6ca7ef459b48a2ba56f788fe37307ee381ced4
Story: 2000488
getProjects() only includes simple job types like Freestyle jobs but
ignores more complex job types like Maven and Matrix jobs. This API is
also marked as deprecated in Jenkins so we probably shouldn't be using
it anyway.
Instead switch to using getAllItems() which pulls down all job types. It
is also possible to pass a parameter such as AbstractProjects or more
specific to specify exactly which job types we want to remove. In this
case though we want all of them so no need to pass any parameters.
See:
http://javadoc.jenkins-ci.org/jenkins/model/Jenkins.html
Change-Id: Ia4e37bc1d4bde70cf8f83783edd6b861eed6c2e3
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
Make sure to perform wrap external to loop, as otherwise this will
recursively wrap the output object and potentially result in getattr
lookups exceeding the recursion limit in python.
Change-Id: Ie9318a3d23a721e62c16e7b05f52d5b9bec1bb82
* Only update_jobs uses the parallel features right now
* --workers N
- If set to 0, it will use parallel execution and
use the number of cores in the machine as thread count
- Any other value enables the parallel extensions and sets the
number of concurrent threads to that value
- Will use 1 by default
Add some tests to make sure the parallel execution helpers work as
expected.
Change-Id: Ib0abd34ea7525f75fff4ff480287a6e589deba90
Signed-off-by: David Caro <dcaroest@redhat.com>
Change Ie952617a34c0719e forced utf-8 format but forgot to decode the
xml when passing to update_job. This broke jjb for python 3:
File "/Users/khaido/PycharmProjects/jenkins-job-builder
/jenkins_jobs/builder.py", line
134, in update_job
self.jenkins.create_job(job_name, xml)
File "/Users/khaido/PycharmProjects/jenkins-job-builder/.tox
/py34/lib/python3.4/site-packages/jenkins/__init__.py",
line 852, in create_job
config_xml.encode('utf-8'), DEFAULT_HEADERS))
AttributeError: 'function' object has no attribute 'encode'
Change-Id: Iceda46214bf4709ccd8141ef654cf3ec81e8af06
Change Id7f7dfb567997e0 introduced a bug where JJB thought that the jobs
were still on the Jenkins server after using the deleting-all command.
Subsequent updates would not re-create the jobs. This change fixes that
problem by clearing out the JJB cache after all jobs have been removed.
Change-Id: I37e65a78f4ba8b05d76b74ff9841d45de9b5bfad
Ensure file objects including stdin/stdout objects are wrapped using
codecs to handle unicode translation to the selected encoding for
input/output.
Add tests to simulate different encodings for input/output and
consequently fix the reading of input from stdin. Include test to
trigger failure to encode a unicode valid character using 'ascii'
encoding.
Change-Id: I9a5c4c96d131873db0000377729b8b0a45aa470d
There is a possible TOCTOU race condition in the event that two
jenkins-jobs instances are running at the same time and try to create
the cache directory at the same time.
story: 2000374
Change-Id: Ie194eb7b52237356e8ab935aed97af577a96605b
Distinguish between ignoring kept managed jobs and unmanaged jobs when
deleting old jobs from the master.
Change-Id: I41e809a775ec9d07863b7c532c98f2eac4c8e5bf
Change If84778231b provided an option to set a connection timeout
in the python-jenkins library. This change allows JJB to override
that timeout value.
Add tests to ensure that the timeout is only set when specified via the
config options.
Change-Id: I3dfe9139469dd0e8549eeedb1833c55ac79ea8b5
delete-all command was deleting one job at a time using the jenkins
doDelete rest endpoint which is very slow. Change Ia4fbfca970165
allows us to execute a groovy script on the jenkins server to
do the same thing. The only difference is that running the script
on the server will delete all jobs much faster.
As a data point:
I added all (~5,000) openstack jobs to my jenkins server.
The previous delete_all took many hours to delete all the
jobs while the script method did it in a matter of seconds.
Change-Id: Id7f7dfb567997e042fe0f783e54680482fefdc15
depends-on: Ia4fbfca970165d890d7e076f47ddcde7633afa9b
Use io.open() to allow reading and writing of files in 'utf-8' format
irrespective of the terminal encoding selected.
Change-Id: Ie952617a34c0719efc59a7729d698beafaa477b0
Also removed the extra return value from the update_jobs method.
Change-Id: Ide838ec5313f736f7a9b4e696f82beb3d9805d57
Signed-off-by: David Caro <dcaroest@redhat.com>
Instead of doing an individual api call each time,
to check if a job exists, do an initial jenkins get_jobs()
call on init, and use that list as a cache, to speed
up job queries.
If a job is not found in cache, we still will be querying
jenkins directly as a fallback.
Change-Id: Ie3eff875f4d832a42c9e38478f62055bc135a7e0
This reverts commit 7100a7f225e60936f23d24c757c1b14c9b136ec4.
The referenced commit, in addition to what it advertises, also
filters the result of update_jobs to include only changed jobs.
If there are no change jobs, then, if the user specifies
--delete-old, there are no jobs in the list of jobs to keep
which comes from the result of update_jobs. In that case, all
jobs are deleted. Or in other words, the logic in this change
is that all jobs not updated are deleted.
Proposing as a revert because this is currently causing serious
problems for continuously deployed JJB instances, and the very
common --delete-old code path should be properly tested with
this change before it lands again.
Change-Id: I98443f0c085e27ed8dfece6409434015ac24b306
With the reorganization of code to move the XmlJob class to a separate
file, the patch to modify the minidom method writexml for earlier
instances that incorrectly added whitespace should be relocated to be
alongside.
This ensures the patch exists along with the code that is using
minidom's functionality.
Change-Id: I6ed3267c3c076dc5f8d40f0c97706c33e4018261
* Only update_jobs uses the parallel features right now
* --workers N
- if set to 0, it will use parallel execution and
use the number of cores in the machine as thread count
- Any other value enables the paralllel extensions and sets the
number of concurrent threads to that value
- Will use 1 by default
Added some tests to make sure the parallel execution helpers work as
expected
Change-Id: Iee8426ff2dc4c5b4ebec4c15bdd21cae5b47eb83
Signed-off-by: David Caro <dcaroest@redhat.com>
The goal of this patch is simply to move some classes out of
jenkins_jobs.builder into more appropriately-named modules. This started with
simply moving YamlParser into jenkins_jobs.parser but led to other moves in
order to avoid cyclic imports since YamlParser uses other classes previously
defined in jenkins_jobs.builder.
That said, this patch doesn't intend to address all of the clutter in
jenkins_jobs.builder, mostly just what is necessary to get started working on
YamlParser independent of other classes in that module.
Change-Id: Ie88bf683e495033eb0b670fe29c256a70282735f
Remove unnecessary calls to matches() and ensure that delete_job() is
correctly passing the jobs_glob argument to the expandYaml() method call
since it uses a string whereas update_job() is using a sequence of
strings.
Once the bug with how delete_job is calling expandYaml() is fixed, the
subsequent call to matches() can be removed.
Change-Id: Iaf1404e26691384fd1d55ea630457860cc1beff5
Added a new configuration option under the section "job_builder" named
"allow_empty_variables" that if set to true, will replace non existing variables
in strings with the empty string instead of raising an error.
It's very useful if you have a shell script, that has optional values, for
example:
EXTRA_PACKAGES=({extra-packages})
for package in "${EXTRA_PACKAGES[@]}"; do
install "$package"
done
Or modifying the script behavior with flags, with empty as default value:
WITH_AUTOGEN={with-autogen}
if [[ $WITH_AUTOGEN ]]; then
./autogen.sh
fi
Then if you have two different jobs that use that script in their builder, you
just have to set the extra parameter or ignore it to change the behavior:
- builder:
name: mybuilder
builders:
- shell: !include shell-scripts/myscript.sh
- job-template:
name: 'mytpl-{name}'
...
builders:
- mybuilder
- project:
name: myproj1
with-autogen: true
jobs:
- 'mytpl-{name}'
- project:
name: myproj2
extra-packages: |
extrapkg1
extrapkg2
Change-Id: Iad9f0e522725e6fd6681cd62d3e36f69baf09585
Signed-off-by: David Caro <dcaroest@redhat.com>
When JJB write jobs to an output directory, each job would attempt to
mkdir the output directory causing unneccessary system calls:
mkdir("output-parent", 0777) = -1 EEXIST (File exists)
stat("output-parent", {st_mode=S_IFDIR|0755, ...}) = 0
Create the output directory before the write loop unless it the output
is writable (stdout) or it already exist. That should speed up the test
operation a little bit.
Change-Id: I17495cba51bd85e4a6f86491a992cec6a3024e72
When user runs 'jenkins-jobs test myjobs.yaml' only the xml
gets printed to screen but the xml doesn't contain any info
about the job name. This logs the job name before showing
the xml snippet for each job. Helps in debugging jobs when
using templates.
For example:
INFO:jenkins_jobs.builder:Job name: foojob-py27-unit-master
<?xml version="1.0" encoding="utf-8"?>
<project>
<actions/>
<description><!-- Managed by Jenkins Job Builder --></description>
<keepDependencies>false</keepDependencies>
<blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
<blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
<concurrentBuild>false</concurrentBuild>
<canRoam>true</canRoam>
<properties/>
<scm class="hudson.scm.NullSCM"/>
<builders>
<hudson.tasks.Shell>
<command>git checkout master</command>
</hudson.tasks.Shell>
</builders>
<publishers/>
<buildWrappers/>
</project>
..
..
Change-Id: Ibce45debd5fd6aa1a400e1cc3762846415ca2a75
Use sphinx's linkcheck builder to validate any external links.
Fix the broken links by converting the example link to a code segment
and correcting the link referencing the cgit mirror for the git clone
instructions.
Change-Id: Ic6236e250b7af5c7c000af1e67fb69dd720ad6d8
This bug occurs if one attempts to set a list object on a job template for a
parameter that does not occur in the job name. If the parameter is not in the
template name then there is no reason to count it among the dimensions used to
fanout that template into multiple instances and in fact this would actually
lead to confusing duplicate job error messages.
Change-Id: I80f026d3bbfbac96c6fc7c01c1c916cf85e1bf10