Home > jenkins > Polling must die: triggering Jenkins builds from a git hook

Polling must die: triggering Jenkins builds from a git hook

December 1st, 2011

As I keep saying, polling a repository from Jenkins is inefficient; it adds delay on the order of minutes before a build starts after a commit is pushed, and it adds additional loads. It is much better instead to do push-notification from the repository. In this post, I’m going to explain how to do this for Git, which brings this on par with Subversion.

The previous best practice of doing this is best summarized in this blog post. While this works, this is less than ideal. A part of the problem is that this requires hard coding of job names inside the repository hooks, making it hard to keep them up-to-date. Another problem is that if your job only cares about one branch in a busy repository, you don’t want a new build to be triggered. Finally, the last problem is that you need some extra work for secured Jenkins.

With the latest Git plugin 1.1.14 (that I just release now), you can now do this more easily by simply executing the following command:

curl http://yourserver/jenkins/git/notifyCommit?url=<URL of the Git repository>

This will scan all the jobs that’s configured to check out the specified URL, and if they are also configured with polling, it’ll immediately trigger the polling (and if that finds a change worth a build, a build will be triggered in turn.) This allows a script to remain the same when jobs come and go in Jenkins. Or if you have multiple repositories under a single repository host application (such as Gitosis), you can share a single post-receive hook script with all the repositories. Finally, this URL doesn’t require authentication even for secured Jenkins, because the server doesn’t directly use anything that the client is sending. It runs polling to verify that there is a change, before it actually starts a build.

One more final note — in Git, unlike Subversion, a repository does not have its own identity, and a single repository sometimes have multiple URLs to access it. In such a case, simply execute the curl commands multiple times with all the different URLs.

That’s it. I hope this will help reduce the use of polling and have more people switch to push notifications. It really is addictive to see the build start in a split second after you push a change.

jenkins , ,

  1. Mark Waite
    August 25th, 2012 at 06:00 | #1

    @Jay This extension was only made to the Jenkins code base, so it won’t work with Hudson. Switch to Jenkins for the latest and most active continuous integration developments.

  2. mns
    August 29th, 2012 at 13:31 | #2

    Can this work with GitHub as well ? Or do I have to use the GitHub Plugin
    to do something similar if I have GitHub Enterprise internally at work ?

  3. August 30th, 2012 at 11:48 | #3

    @mns
    No, GitHub doesn’t allow you to directly touch post-receive hook. You should use the GitHub plugin for it, and it should work with GitHub Enterprise, too.

  4. August 30th, 2012 at 13:11 | #4

    @eedri
    I doubt if this works with multi SCM plugin. “Multi SCM” feature is something we really need to bring back into the core.

  5. sroth80021
    September 8th, 2012 at 23:50 | #5

    Unfortunately, I can verify this does not work with the MultiSCM plugin.

    I’ve read there were plans to rewrite the git plugin to natively support multiple git repos, but given the last comment date was quite awhile ago, I’m wondering if this is happening or not. Or maybe I’m just out-of-the-loop here.

  6. October 1st, 2012 at 06:09 | #6

    I am having trouble executing the build with this trigger in timely manner.
    Namely, the trigger is executed, and the job is also found (and is valid / scheduled) but the build occurs about 10-20 minutes later.

    I’ve done a bit of investigating and this is the flow and the information I can gather from logs:

    * the project (e.g. jenkins-hooks-test) has a git(lab) repository, with the following web hook: http://ci-server/jenkins/git/notifyCommit?url=http://git.infobip.local/jenkins-hooks-test

    * the project has received a push to the repository

    * I can see that the hook fired the notifyCommit on jenkins, this is the trigger log:
    01.10.2012. 14:46:46 hudson.triggers.SCMTrigger
    FINE: Scheduling a polling for hudson.maven.MavenModuleSet@115be90[jenkins-hooks-test]

    * the build starts 7 minutes later, the polling log:
    Started on 01.10.2012. 14:53:00
    Using strategy: Default
    [poll] Last Build : #22
    [poll] Last Built Revision: Revision 74a83490028ebf3b15af36a90d443f98a5e8262c (origin/master)
    Fetching changes from the remote Git repositories
    Fetching upstream changes from git@git.infobip.local:jenkins-hooks-test.git
    Polling for changes in
    Done. Took 32 sec
    Changes found

    So, from the chain of events point of view, this all OK since a change in the repository fires the build/polling trigger. The only problem I have is the delay. I’ve tested this when Jenkins is loaded with jobs but also when it’s building nothing.

    The project is configured to Poll SCM @yearly and Build whenever a SNAPSHOT dependency is built. No other build triggers are present.

    The main Jenkins configuration has the Quiet period set to 0.
    I am using Jenkins 1.483 and git plugin 1.1.24.

    Jenkins is configured with about 260 projects, of which most is currently using scheduled polling (every minute usually). I counted these by viewing all projects on the dashboard since the following snippet returns 1252 projects:
    println Hudson.getInstance().getAllItems(AbstractProject.class).size()
    (can’t figure out why the numbers don’t match)

    I’ve also tried executing the trigger with this url: git@git.infobip.local:jenkins-hooks-test.git, and I get the same behaviour.

    If used from curl, the response is:
    “Scheduled polling of jenkins-hooks-test”
    which notes that the build/polling trigger performed OK
    (from what I can see in plugin code).

    The only thing that is suspicious is probably the isStarving() on the trigger.
    I’ve used the below script to get the information and it is in fact starving for 5 minutes (even more). I’ve also used the same script to run the trigger manually, with the same effect.

    import hudson.plugins.git.*
    import hudson.triggers.*

    for (AbstractProject project : Hudson.getInstance().getAllItems(AbstractProject.class)) {
    if(project.name.equals(“jenkins-hooks-test”)) {
    println “project ${project.name}”
    SCMTrigger trigger = project.getTrigger(SCMTrigger.class)
    //println “executing the build trigger on the project”
    //trigger.run()
    println “the trigger is starving: ${trigger.descriptor.queue.isStarving(5 * 60 * 1000)}”
    }
    }

    Any help appreciated!

    P.S. If there is a smarter way to use gitlab with this plugin, I’d be very happy to give a hand…

  7. October 3rd, 2012 at 06:15 | #7

    P.P.S. the following starts the build instantly…
    Maybe this is OK, since we are in fact “notifying of change to be built” ?

    import hudson.plugins.git.*
    import hudson.triggers.*
    import hudson.model.*

    for (AbstractProject project : Hudson.getInstance().getAllItems(AbstractProject.class)) {
    if(project.name.equals(“jenkins-hooks-test”)) {
    println “building project ${project.name}”
    project.scheduleBuild(new Cause.RemoteCause(“gitlab”, “changes in the build”))
    }
    }

  8. October 22nd, 2012 at 05:18 | #8

    @Vanja Radovanović
    I’ve created a gitlab hook plugin that offers both notify commit and build now options. This is not a direct solution but it works for us. You can find details @ https://wiki.jenkins-ci.org/display/JENKINS/Gitlab+Hook+Plugin

  9. Kelevra
    December 18th, 2012 at 02:10 | #9

    Hi. Can You tell me, please, where I need to add this command in jenkins?

  10. Ryan
    January 17th, 2013 at 14:23 | #10

    How do I get this to work with all users of the repository? Git hooks appear to be user specific and are not saved in the repository.

  11. Erik
    January 21st, 2013 at 06:51 | #11

    I could not get the correct refs, but this worked for me:

    for ref in $@; do
    IFS=’/’ read -ra REF <<< "${ref}"
    branch="${REF[2]}"
    branch_list=${branch_list}${delim}${branch}
    delim=","
    done

    I think its because we still have a old git version perhaps.

  12. Sumeet
    February 26th, 2013 at 22:37 | #12

    I have the setup working for git branches. Any new change triggers the builds configured.

    @kohsuke
    How do I make it work for any new git tag created manually? (As soon as a new tag is created, the hook causes the builds configured to trigger).

  13. srini chokkarapu
    September 19th, 2013 at 08:54 | #13

    Hi, Kohsuke wrote “This will scan all the jobs that’s configured to check out the specified URL, and if they are also configured with polling, it’ll immediately trigger the polling (and if that finds a change worth a build, a build will be triggered in turn.) “.
    When you say if they also configured with the polling, I guess you meant the Jenkin’s job, right?
    If I don’t configure the Jenkin’s job to poll, then when does it trigger the build? I waited for about 23 mins still it didn’t trigger the build.

  14. Maneesh
    September 30th, 2013 at 17:32 | #14

    How can this work if the repository is placed on AWS and jenkins is running inside a firewall?

  15. JM Kinsley
    March 6th, 2014 at 23:20 | #15

    Either this plugin or Jenkins has a bug where specifying the git url of the form: `ssh://foo@bar.lan.local:7999/baz.git` in `http://yourserver/jenkins/git/notifyCommit?url=` above will give NullPointerException in Jenkins. This tool has too many bugs to be taken seriously (http://issues.jenkins-ci.org/secure/IssueNavigator.jspa?mode=hide&reset=true&jqlQuery=project+%3D+JENKINS+AND+status+in+%28Open%2C+%22In+Progress%22%2C+Reopened%29+AND+component+%3D+%27git%27)

Comment pages
  1. December 20th, 2011 at 22:50 | #1
  2. September 25th, 2012 at 02:03 | #2
  3. March 8th, 2013 at 15:16 | #3