Polling must die: triggering Jenkins builds from a git hook

As I keep saying, polling a repository from Jenkins is inefficient; it adds delay on the order of minutes before a build starts after a commit is pushed, and it adds additional loads. It is much better instead to do push-notification from the repository. In this post, I’m going to explain how to do this for Git, which brings this on par with Subversion.

The previous best practice of doing this is best summarized in this blog post. While this works, this is less than ideal. A part of the problem is that this requires hard coding of job names inside the repository hooks, making it hard to keep them up-to-date. Another problem is that if your job only cares about one branch in a busy repository, you don’t want a new build to be triggered. Finally, the last problem is that you need some extra work for secured Jenkins.

With the latest Git plugin 1.1.14 (that I just release now), you can now do this more easily by simply executing the following command:

curl http://yourserver/jenkins/git/notifyCommit?url=<URL of the Git repository>

This will scan all the jobs that’s configured to check out the specified URL, and if they are also configured with polling, it’ll immediately trigger the polling (and if that finds a change worth a build, a build will be triggered in turn.) This allows a script to remain the same when jobs come and go in Jenkins. Or if you have multiple repositories under a single repository host application (such as Gitosis), you can share a single post-receive hook script with all the repositories. Finally, this URL doesn’t require authentication even for secured Jenkins, because the server doesn’t directly use anything that the client is sending. It runs polling to verify that there is a change, before it actually starts a build.

One more final note — in Git, unlike Subversion, a repository does not have its own identity, and a single repository sometimes have multiple URLs to access it. In such a case, simply execute the curl commands multiple times with all the different URLs.

That’s it. I hope this will help reduce the use of polling and have more people switch to push notifications. It really is addictive to see the build start in a split second after you push a change.

86 Comments Add yours

  1. PJ says:

    Thank you! That’s a much better solution that what we currently do:

    ref_in_branch() {
    if echo $refname | grep -q $reftomatch ; then
    return 0
    return 1

    build_proj() {
    # cause hudson to build a project
    # the project to build

    # Log a timestamp
    log `date` Building project: $project

    wget -O /dev/null -q “${url}” &

    # get info about the update from stdin
    while read oldrev newrev refname ; do
    ref_in_branch version-5.0 && build_proj project_v5.0
    ref_in_branch version-6.0 && build_proj project_v6.0
    ref_in_branch version-7.0 && build_proj project_v7.0

  2. Maarten Dirkse says:

    This works great, nice feature!
    I have a question though: why is it necessary to configure polling on my jobs in order for this hook to work? For example: I have a job that is set up to clone a repo and build the single project in that repo. I only want this job executed when there is a push to the repo. Using the curl method I can launch a build when a push is done, but I still need to configure a polling schedule, which, with the build-on-push functionality, has just become useless. So I’ve now configured my job to poll once a year (aka never). Wouldn’t it be better to have an option among the build triggers along the lines of “Build is triggered by git hook”, and allow the launching of builds that way?
    Seems a bit ironic that in order to achieve your “polling must die” aim, you still have to configure polling 😉

  3. Maarten Dirkse says:

    If your git URL’s are of the format http://yourgitserver/repo.git then adding the following to your server-side post-update script (in ./repodir/hooks) will trigger any builds for the repo:

    IFS=’;’ read -ra REPOPATH << /dev/null

    The snippet figures out the name of the repo by getting the name of the parent directory of the directory that the post-update script is located in.

  4. Maarten Dirkse says:

    If your git URL’s are of the format http://yourgitserver/repo.git then adding the following to your server-side post-update script (in ./repodir/hooks) will trigger any builds for the repo:

    IFS=’;’ read -ra REPOPATH <<< “$( cd “$( dirname “${BASH_SOURCE[0]}” )” && pwd | tr / ‘;’ | cut -c2-)”
    curl http://jenkins.socs.lan/git/notifyCommit?url=http://$HOSTNAME/${REPOPATH[${#REPOPATH}-1]} > /dev/null

    The snippet figures out the name of the repo by getting the name of the parent directory of the directory that the post-update script is located in.

  5. kohsuke says:

    @Maarten Dirkse
    We require the polling configuration as a sign that you want this job be built on code changes. This allows you to have jobs that are triggered from other reasons but still uses the same code base, such as a release job, etc.

    Yes, I agree that asking the user to fill in a meaningless cron value is bad, and I’ve heard other people argue for having the generic “Build is triggered by some push mechanism” option that all other SCMs can use as the marker.

    The downside is that novice users might incorrectly think that just checking it would magically make the whole thing work, when the push setup is still needed. But I guess we should probably do it nonetheless, and have the help clearly explain steps to make it work.

  6. Maarten Dirkse says:

    I understand your concern, but I think with a properly worded option, and good help, like you said, novice users will get it. And I think it’ll be more novice-friendly than the current situation of having to set up polling in order to get push to work. Although with all the political primaries that are about to happen, a “push poll” feature might be timely 😛

    I was thinking something along the lines of this: http://yfrog.com/z/ken9qp
    If I get some time today or this weekend, I’ll take a crack at implementing it myself.

  7. Yecine says:


    I’m trying to setup this but i can’t get it to work … i always have no git jobs found

    here is the command that i have

    curl http://server:8080/jenkins/git/notifyCommit?url=ssh://git@repo.com/myfirstproject.git

    I have setup polling with 0 10 * * *

    what did i do wrong ?

  8. kohsuke says:

    @Maarten Dirkse
    Great. The only change I’d make is to remove the need for the user to select SCM. As we talked in Twitter, that is unnecessary.

  9. kohsuke says:

    Increase the logging level for hudson.plugins.git.GitStatus and see what it’ll report.

  10. Mike Henke says:

    I get “Jobs found but they aren’t configured for polling” when going directly to the url

    Is this correct?

  11. Mike Henke says:

    I when Poll SCM and check that, then set it to 5 * * * *. Then got the message “Scheduled polling of Rewrite” when running the url.

  12. marlene cote says:

    If this calls all the jobs that are configured to poll, then we will continue to see the problem we are seeing now. We are using the git plugin with jenkins. When a job polls, it sometimes gets the wrong previous version built, and kicks off a job that has nothing to do.
    It spends time creating a workspace and populating submodules, etc and then exits. This is a waste of computing resources, and if there is a real change it is delayed by these bogus builds. How are you addressing the “real” problem with polling?

  13. marlene cote says:

    This command also causes polling to occur on disabled jobs. Could you enhance it so that doesn’t happen? And is it possible for me to pick the job that I want to poll??? Thanks.

  14. kohsuke says:

    @Mike Henke
    Yep, so you got it working correctly!

  15. kohsuke says:

    @marlene cote
    What this internally does it to trigger the polling in reaction to the notifcation, so I don’t think this will affect the problem you are describing.

    In 1.1.15, I’ve fixed the handling of disabled jobs.

  16. Gergo says:

    Sounds great, thank you.

    By migrating to this hook, can I reduce the number of threads in Jenkins? (last time I checked, I had around 50 of them :-/ and getting some freezes too, so wondering if these could be related to polling).

    Also, on the issue guys above raised: what shall I set the (fake) polling schedule to? IOW, how does Jenkins actually distinguish between the fake polling scenario and the real ones – as there is no extra configuration needed will real polling still work (for non-git or non-hooked repos).

  17. kohsuke says:

    Can’t say about the # of threads until I know what they are. But you can certainly expect there to be less polling activities, and if those are the threads that are hanging around, you can expect less of them.

    The fake polling schedule could be anythng. Say @yearly. Jenkins does follow this schedule, so in that sense Jenkins doesn’t distinguish fake polling from real polling. It’s just that the activation of the polling features serves as a filter for the push notification receiver code.

  18. Gergo says:

    @kohsuke Thanks for the prompt and helpful reply. Would blank ” be a good fake schedule? Seems to be accepted – as opposed to -1 or 60 for minute value.

  19. kohsuke says:

    Didn’t realize we accept blank, but if we do, sure.

  20. Jason says:

    Our developers usually push their changes to remote repo and then make a new tag to identify that for release. In this case, the second push won’t cause the build running as there are actually no new changes between two pushes. Is this the way in which it’s designed? Thanks.

  21. kohsuke says:

    Yes. As you say, there’s no change in the code. Or am I missing the point?

  22. Jason says:

    I see. I’ve added a new GIT hook to trigger a specific Jenkins project directly whenever a new change is submitted for our special case, no matter it’s code change or just a tag.

  23. Shak says:

    I don’t understand why we need polling set up – shouldn’t this avoid that? Should I set my job with a vacuously long polling period?

  24. Senthil says:

    I have one problem with this. When a job is configured to have Polling with a dummy cron, if the job is triggered by other means the Polling Log link says “Polling has not run yet.” Does this mean that for the Polling mechanism to work (i.e. to Poll changes and setup variables like “${CHANGES_SINCE_LAST_SUCCESS}”) is for the cron job to actually start on it’s own?

  25. kohsuke says:

    Yes, set it up with a long polling cycle. This is needed because some jobs aren’t intended to be driven by changes in the repository. The activation of polling indicates that you intend to let changes in source code kick builds.

  26. kohsuke says:

    It is an annoyance indeed but it doesn’t have any negative implications.

  27. Senthil says:

    I think I disagree. Suppose that the build is manually triggered, it should not build if there are no new changes since the last build, right? Also, if the build is not triggered by Polling, then some of the Polling variables are not set and this is very misleading.

    My particular problem is that I want to use “${CHANGES_SINCE_LAST_SUCCESS}” in an auto-generated e-mail. But if Polling does not start the build, this variable is not set, and a successful build e-mail is sent, but it shows zero changes.

    I have it configured as you mentioned, but I don’t see a solution to the problem I stated. I would like to use Polling to help auto-generate this kind of e-mail. But it looks like Polling kicks off a build, but a build is unable to start Polling. Should I be doing this differently?

  28. Senthil says:

    I take back what I wrote. Something else is wrong, even if Polling triggers the build, I’m not getting any changes shown in the completion e-mail. I had it working when Polling kicked off the build, but that’s not working anymore. I’ll investigate further.

  29. Senthil says:

    Ok… so I did a lot of trial and error, and finally determined that if you change the config of a job, the polling results somehow get lost and then next build will always say “Failed to determine (log)” in the Changes page. So, you have to make your config changes, and start 2 builds to see Polling work and have the job correctly determine changes since the last build.

  30. Senthil says:

    Sorry to derail this thread off-track! ^^;

  31. Lee says:

    We have over 100 builds that get scheduled for polling with the current functionality. Most of these are set to poll older maintenance branches that are rarely updated. The post-update hook gets the list of branches that were updated in a push. If that list was sent along, would it be a good idea to also only trigger polling for builds that had matched one of the given branches?

    curl http://yourserver/jenkins/git/notifyCommit?url=&branch=&branch=

    The post-update could look like this:

    for ref in $@; do
    branch=”&branch=$(git rev-parse –symbolic –abbrev-ref $ref)”
    echo curl https://yourserver/jenkins/git/notifyCommit?url=${branches}

  32. kohsuke says:

    Ah, that’s interesting. Yes, that’d be definitely useful. We should add it (or better yet, would you be interested in adding it :-)?

  33. Lee says:

    @kohsuke I was thinking of doing a pull request with it. It’s just a matter of getting a test setup for it… I haven’t done Java in a long time. I’ll give it a shot.

  34. Gavin says:

    I am so happy this was implemented. No more polling constantly for changes. Its so easy to do a “curl -S” with ${GL_REPO} using gitolite.

  35. Alex says:

    Does Jenkins have functionality to poll on startup? I am thinking of the scenario when Jenkins is down for whatever reason. Developers continue checking code in. Jenkins starts and awaits the next commit. It would be nice to have the option to poll on startup in case there was a missed commit trigger received.

  36. Lee says:

    @kohsuke, I created a pull request that only polls builds that work on the updated branches. https://github.com/jenkinsci/git-plugin/pull/56

  37. Peter Kline says:

    I have an implementation question. We see on a rare occasion a double build where the same sha1 is built twice. I think it may have something to do with two pushes done very close together.

    [poll] Last Build : #1417
    [poll] Last Built Revision: Revision 77ccf9a674542eb6d531e90402dcbd58bd35fcb5 (remotes/origin/main_int)

    [poll] Last Build : #1418
    [poll] Last Built Revision: Revision 77ccf9a674542eb6d531e90402dcbd58bd35fcb5 (remotes/origin/main_int)

    These polling triggers were done three minutes apart. Is there something we can do where the same sha1 isn’t triggered for a build?

  38. David Kinzer says:

    Thanks! This was very useful (especially because we are behind a firewall and it’s a challenge to set up builds by having GitHub report to our Jenkins server directly).

  39. Chris Hansen says:

    This is seriously cool. Thanks for a great plugin.
    I can confirm that leaving the schedule empty works just as well as putting in something bogus. However, I do agree that a tag (e.g. @Triggered), or perhaps a checkbox removing the schedule area if checked, would be very useful.

  40. Scot Wilkie says:

    I am wondering if my git plugin is installed incorrectly. I do not have a git directory under jenkins, so the path http://yourserver/jenkings/git/notifyCommit is not found. I do have a git directory under plugins and I can run jobs which clone repos, but I can’t find a path to notifyCommit.

    I am new to jenkins so any help is greatly appreciated.

  41. Scot Wilkie says:

    @Scot Wilkie
    My git plugin was version 1.1.12 and there were no updates listed under the update tab. I manually down loaded 1.1.18 and copied it to the pulgins directory and restarted Jenkins. The Git directory seems to have the correct date/time stamp of 4/27/2012, but in the installed plugins tab it still reports the git plugin as version 1.1.12.

  42. Scot Wilkie says:

    I upgraded to the latest Jenkins version and then the git plugin was correctly identified as 1.1.18. However now when I run my jobs I get an error about not finding any version to build. I down graded to version 1.1.14 and the jobs run fine. Please let me know if you need any information about my setup.

  43. Scot Wilkie says:

    Sorry, I didn’t mean to hijack the thread with my comments on the git plugin versions (but 1.1.18 does not work at all for me). After updating Jenkins to the latest version today, 1.463, and also installing the Jenkins Notification plug in (because it sounds like it is needed) things sort of work. I have 3 Jenkins jobs, each one to build on master, br2, and br3 respectively. Doing a push now shows 3 lines in STDOUT indicating that scheduled polling was started for each of the 3 branches. However only a push to br2 seems to trigger a build on Jenkins, and then the jobs for both br2 and br3 run; pushes on master or br3, while getting the same STDOUT lines about scheduled polling, do not trigger the corresponding jobs to start. Seems like I am missing some criteria of what it takes to trigger the Jenkins job to start. Here is the line in my post-receive hook for the repo:
    curl http://smfosbuild:8080/git/notifyCommit?url=git@vfilvgit2:scmtest.git

    Any ideas?

  44. David Kinzer says:

    To me it makes sense to poll. All that hook does is trigger on a POST to a URL. I don’t want the build to occur just because some bot or prankster is posting to that URL for kicks.

    Polling to see that a change has actually occurred makes it less likely to get frivolous builds. @Maarten Dirkse

  45. Khang says:

    Add the git URL after url= for the comment right above.

  46. Kieran says:

    Thanks a lot, really useful to know we can do this. I’m fed up of having so many jobs polling all the time!

  47. eedri says:

    we’re using mainly multiple SCM plugin and not the standard git plugin.
    should this work with it as well?

    when i tried running the command it said:
    No git jobs using repository: …

    while actually there were some that did use it with the multiple SCM plugin

  48. Jay says:

    I just get a 404 when I visit the URL (stand-alone version). I’m using hudson (2.2.1) not jenkins, does that matter? Git plugin is version 2.2.0. The URL I’m trying is like this:


Leave a Reply

Your email address will not be published. Required fields are marked *