Polling must die: triggering Jenkins builds from a git hook

As I keep saying, polling a repository from Jenkins is inefficient; it adds delay on the order of minutes before a build starts after a commit is pushed, and it adds additional loads. It is much better instead to do push-notification from the repository. In this post, I’m going to explain how to do this for Git, which brings this on par with Subversion.

The previous best practice of doing this is best summarized in this blog post. While this works, this is less than ideal. A part of the problem is that this requires hard coding of job names inside the repository hooks, making it hard to keep them up-to-date. Another problem is that if your job only cares about one branch in a busy repository, you don’t want a new build to be triggered. Finally, the last problem is that you need some extra work for secured Jenkins.

With the latest Git plugin 1.1.14 (that I just release now), you can now do this more easily by simply executing the following command:

curl http://yourserver/jenkins/git/notifyCommit?url=<URL of the Git repository>

This will scan all the jobs that’s configured to check out the specified URL, and if they are also configured with polling, it’ll immediately trigger the polling (and if that finds a change worth a build, a build will be triggered in turn.) This allows a script to remain the same when jobs come and go in Jenkins. Or if you have multiple repositories under a single repository host application (such as Gitosis), you can share a single post-receive hook script with all the repositories. Finally, this URL doesn’t require authentication even for secured Jenkins, because the server doesn’t directly use anything that the client is sending. It runs polling to verify that there is a change, before it actually starts a build.

One more final note — in Git, unlike Subversion, a repository does not have its own identity, and a single repository sometimes have multiple URLs to access it. In such a case, simply execute the curl commands multiple times with all the different URLs.

That’s it. I hope this will help reduce the use of polling and have more people switch to push notifications. It really is addictive to see the build start in a split second after you push a change.

86 thoughts on “Polling must die: triggering Jenkins builds from a git hook”

  1. Thank you! That’s a much better solution that what we currently do:

    ref_in_branch() {
    reftomatch=$1
    if echo $refname | grep -q $reftomatch ; then
    return 0
    fi
    return 1
    }

    build_proj() {
    # cause hudson to build a project
    # the project to build
    project=$1

    # Log a timestamp
    log `date` Building project: $project

    url=${hudson}/${project}/build
    wget -O /dev/null -q “${url}” &
    }

    # get info about the update from stdin
    while read oldrev newrev refname ; do
    ref_in_branch version-5.0 && build_proj project_v5.0
    ref_in_branch version-6.0 && build_proj project_v6.0
    ref_in_branch version-7.0 && build_proj project_v7.0
    done

  2. This works great, nice feature!
    I have a question though: why is it necessary to configure polling on my jobs in order for this hook to work? For example: I have a job that is set up to clone a repo and build the single project in that repo. I only want this job executed when there is a push to the repo. Using the curl method I can launch a build when a push is done, but I still need to configure a polling schedule, which, with the build-on-push functionality, has just become useless. So I’ve now configured my job to poll once a year (aka never). Wouldn’t it be better to have an option among the build triggers along the lines of “Build is triggered by git hook”, and allow the launching of builds that way?
    Seems a bit ironic that in order to achieve your “polling must die” aim, you still have to configure polling ;)
    Regards,
    Maarten

  3. If your git URL’s are of the format http://yourgitserver/repo.git then adding the following to your server-side post-update script (in ./repodir/hooks) will trigger any builds for the repo:

    IFS=';’ read -ra REPOPATH << /dev/null

    The snippet figures out the name of the repo by getting the name of the parent directory of the directory that the post-update script is located in.

  4. If your git URL’s are of the format http://yourgitserver/repo.git then adding the following to your server-side post-update script (in ./repodir/hooks) will trigger any builds for the repo:

    IFS=';’ read -ra REPOPATH <<< “$( cd “$( dirname “${BASH_SOURCE[0]}” )” && pwd | tr / ‘;’ | cut -c2-)”
    curl http://jenkins.socs.lan/git/notifyCommit?url=http://$HOSTNAME/${REPOPATH[${#REPOPATH}-1]} > /dev/null

    The snippet figures out the name of the repo by getting the name of the parent directory of the directory that the post-update script is located in.

  5. @Maarten Dirkse
    We require the polling configuration as a sign that you want this job be built on code changes. This allows you to have jobs that are triggered from other reasons but still uses the same code base, such as a release job, etc.

    Yes, I agree that asking the user to fill in a meaningless cron value is bad, and I’ve heard other people argue for having the generic “Build is triggered by some push mechanism” option that all other SCMs can use as the marker.

    The downside is that novice users might incorrectly think that just checking it would magically make the whole thing work, when the push setup is still needed. But I guess we should probably do it nonetheless, and have the help clearly explain steps to make it work.

  6. @kohsuke
    I understand your concern, but I think with a properly worded option, and good help, like you said, novice users will get it. And I think it’ll be more novice-friendly than the current situation of having to set up polling in order to get push to work. Although with all the political primaries that are about to happen, a “push poll” feature might be timely :P

    I was thinking something along the lines of this: http://yfrog.com/z/ken9qp
    If I get some time today or this weekend, I’ll take a crack at implementing it myself.

  7. If this calls all the jobs that are configured to poll, then we will continue to see the problem we are seeing now. We are using the git plugin with jenkins. When a job polls, it sometimes gets the wrong previous version built, and kicks off a job that has nothing to do.
    It spends time creating a workspace and populating submodules, etc and then exits. This is a waste of computing resources, and if there is a real change it is delayed by these bogus builds. How are you addressing the “real” problem with polling?

  8. This command also causes polling to occur on disabled jobs. Could you enhance it so that doesn’t happen? And is it possible for me to pick the job that I want to poll??? Thanks.

  9. @marlene cote
    What this internally does it to trigger the polling in reaction to the notifcation, so I don’t think this will affect the problem you are describing.

    In 1.1.15, I’ve fixed the handling of disabled jobs.

  10. Sounds great, thank you.

    By migrating to this hook, can I reduce the number of threads in Jenkins? (last time I checked, I had around 50 of them :-/ and getting some freezes too, so wondering if these could be related to polling).

    Also, on the issue guys above raised: what shall I set the (fake) polling schedule to? IOW, how does Jenkins actually distinguish between the fake polling scenario and the real ones – as there is no extra configuration needed will real polling still work (for non-git or non-hooked repos).

  11. @Gergo
    Can’t say about the # of threads until I know what they are. But you can certainly expect there to be less polling activities, and if those are the threads that are hanging around, you can expect less of them.

    The fake polling schedule could be anythng. Say @yearly. Jenkins does follow this schedule, so in that sense Jenkins doesn’t distinguish fake polling from real polling. It’s just that the activation of the polling features serves as a filter for the push notification receiver code.

  12. @kohsuke Thanks for the prompt and helpful reply. Would blank ” be a good fake schedule? Seems to be accepted – as opposed to -1 or 60 for minute value.

  13. Our developers usually push their changes to remote repo and then make a new tag to identify that for release. In this case, the second push won’t cause the build running as there are actually no new changes between two pushes. Is this the way in which it’s designed? Thanks.

  14. @kohsuke
    I see. I’ve added a new GIT hook to trigger a specific Jenkins project directly whenever a new change is submitted for our special case, no matter it’s code change or just a tag.

  15. I don’t understand why we need polling set up – shouldn’t this avoid that? Should I set my job with a vacuously long polling period?

  16. I have one problem with this. When a job is configured to have Polling with a dummy cron, if the job is triggered by other means the Polling Log link says “Polling has not run yet.” Does this mean that for the Polling mechanism to work (i.e. to Poll changes and setup variables like “${CHANGES_SINCE_LAST_SUCCESS}”) is for the cron job to actually start on it’s own?

  17. @Shak
    Yes, set it up with a long polling cycle. This is needed because some jobs aren’t intended to be driven by changes in the repository. The activation of polling indicates that you intend to let changes in source code kick builds.

  18. @kohsuke
    I think I disagree. Suppose that the build is manually triggered, it should not build if there are no new changes since the last build, right? Also, if the build is not triggered by Polling, then some of the Polling variables are not set and this is very misleading.

    My particular problem is that I want to use “${CHANGES_SINCE_LAST_SUCCESS}” in an auto-generated e-mail. But if Polling does not start the build, this variable is not set, and a successful build e-mail is sent, but it shows zero changes.

    I have it configured as you mentioned, but I don’t see a solution to the problem I stated. I would like to use Polling to help auto-generate this kind of e-mail. But it looks like Polling kicks off a build, but a build is unable to start Polling. Should I be doing this differently?

  19. @Senthil
    I take back what I wrote. Something else is wrong, even if Polling triggers the build, I’m not getting any changes shown in the completion e-mail. I had it working when Polling kicked off the build, but that’s not working anymore. I’ll investigate further.

  20. @Senthil
    Ok… so I did a lot of trial and error, and finally determined that if you change the config of a job, the polling results somehow get lost and then next build will always say “Failed to determine (log)” in the Changes page. So, you have to make your config changes, and start 2 builds to see Polling work and have the job correctly determine changes since the last build.

  21. We have over 100 builds that get scheduled for polling with the current functionality. Most of these are set to poll older maintenance branches that are rarely updated. The post-update hook gets the list of branches that were updated in a push. If that list was sent along, would it be a good idea to also only trigger polling for builds that had matched one of the given branches?

    curl http://yourserver/jenkins/git/notifyCommit?url=&branch=&branch=

    The post-update could look like this:

    #!/bin/sh
    branches=””
    for ref in $@; do
    branch=”&branch=$(git rev-parse –symbolic –abbrev-ref $ref)”
    branches=${branches}${branch}
    done
    echo curl https://yourserver/jenkins/git/notifyCommit?url=${branches}

  22. @kohsuke I was thinking of doing a pull request with it. It’s just a matter of getting a test setup for it… I haven’t done Java in a long time. I’ll give it a shot.

  23. I am so happy this was implemented. No more polling constantly for changes. Its so easy to do a “curl -S” with ${GL_REPO} using gitolite.

  24. Does Jenkins have functionality to poll on startup? I am thinking of the scenario when Jenkins is down for whatever reason. Developers continue checking code in. Jenkins starts and awaits the next commit. It would be nice to have the option to poll on startup in case there was a missed commit trigger received.

  25. I have an implementation question. We see on a rare occasion a double build where the same sha1 is built twice. I think it may have something to do with two pushes done very close together.

    [poll] Last Build : #1417
    [poll] Last Built Revision: Revision 77ccf9a674542eb6d531e90402dcbd58bd35fcb5 (remotes/origin/main_int)

    [poll] Last Build : #1418
    [poll] Last Built Revision: Revision 77ccf9a674542eb6d531e90402dcbd58bd35fcb5 (remotes/origin/main_int)

    These polling triggers were done three minutes apart. Is there something we can do where the same sha1 isn’t triggered for a build?
    Thanks

  26. Thanks! This was very useful (especially because we are behind a firewall and it’s a challenge to set up builds by having GitHub report to our Jenkins server directly).

  27. This is seriously cool. Thanks for a great plugin.
    I can confirm that leaving the schedule empty works just as well as putting in something bogus. However, I do agree that a tag (e.g. @Triggered), or perhaps a checkbox removing the schedule area if checked, would be very useful.

  28. I am wondering if my git plugin is installed incorrectly. I do not have a git directory under jenkins, so the path http://yourserver/jenkings/git/notifyCommit is not found. I do have a git directory under plugins and I can run jobs which clone repos, but I can’t find a path to notifyCommit.

    I am new to jenkins so any help is greatly appreciated.

  29. @Scot Wilkie
    My git plugin was version 1.1.12 and there were no updates listed under the update tab. I manually down loaded 1.1.18 and copied it to the pulgins directory and restarted Jenkins. The Git directory seems to have the correct date/time stamp of 4/27/2012, but in the installed plugins tab it still reports the git plugin as version 1.1.12.

  30. I upgraded to the latest Jenkins version and then the git plugin was correctly identified as 1.1.18. However now when I run my jobs I get an error about not finding any version to build. I down graded to version 1.1.14 and the jobs run fine. Please let me know if you need any information about my setup.

  31. Sorry, I didn’t mean to hijack the thread with my comments on the git plugin versions (but 1.1.18 does not work at all for me). After updating Jenkins to the latest version today, 1.463, and also installing the Jenkins Notification plug in (because it sounds like it is needed) things sort of work. I have 3 Jenkins jobs, each one to build on master, br2, and br3 respectively. Doing a push now shows 3 lines in STDOUT indicating that scheduled polling was started for each of the 3 branches. However only a push to br2 seems to trigger a build on Jenkins, and then the jobs for both br2 and br3 run; pushes on master or br3, while getting the same STDOUT lines about scheduled polling, do not trigger the corresponding jobs to start. Seems like I am missing some criteria of what it takes to trigger the Jenkins job to start. Here is the line in my post-receive hook for the repo:
    curl http://smfosbuild:8080/git/notifyCommit?url=git@vfilvgit2:scmtest.git

    Any ideas?

  32. To me it makes sense to poll. All that hook does is trigger on a POST to a URL. I don’t want the build to occur just because some bot or prankster is posting to that URL for kicks.

    Polling to see that a change has actually occurred makes it less likely to get frivolous builds. @Maarten Dirkse

  33. Thanks a lot, really useful to know we can do this. I’m fed up of having so many jobs polling all the time!

  34. we’re using mainly multiple SCM plugin and not the standard git plugin.
    should this work with it as well?

    when i tried running the command it said:
    No git jobs using repository: …

    while actually there were some that did use it with the multiple SCM plugin

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>