As I keep saying, polling a repository from Jenkins is inefficient; it adds delay on the order of minutes before a build starts after a commit is pushed, and it adds additional loads. It is much better instead to do push-notification from the repository. In this post, I’m going to explain how to do this for Git, which brings this on par with Subversion.
The previous best practice of doing this is best summarized in this blog post. While this works, this is less than ideal. A part of the problem is that this requires hard coding of job names inside the repository hooks, making it hard to keep them up-to-date. Another problem is that if your job only cares about one branch in a busy repository, you don’t want a new build to be triggered. Finally, the last problem is that you need some extra work for secured Jenkins.
With the latest Git plugin 1.1.14 (that I just release now), you can now do this more easily by simply executing the following command:
curl http://yourserver/jenkins/git/notifyCommit?url=<URL of the Git repository>
This will scan all the jobs that’s configured to check out the specified URL, and if they are also configured with polling, it’ll immediately trigger the polling (and if that finds a change worth a build, a build will be triggered in turn.) This allows a script to remain the same when jobs come and go in Jenkins. Or if you have multiple repositories under a single repository host application (such as Gitosis), you can share a single post-receive hook script with all the repositories. Finally, this URL doesn’t require authentication even for secured Jenkins, because the server doesn’t directly use anything that the client is sending. It runs polling to verify that there is a change, before it actually starts a build.
One more final note — in Git, unlike Subversion, a repository does not have its own identity, and a single repository sometimes have multiple URLs to access it. In such a case, simply execute the curl commands multiple times with all the different URLs.
That’s it. I hope this will help reduce the use of polling and have more people switch to push notifications. It really is addictive to see the build start in a split second after you push a change.
Nice article and its very clear on how to do a polling based on git from Jenkins.
Is there any feature requested in future versions of git plugin to add git command executions in the Build phase or Post build actions phase(like merge after build)?
@Jay This extension was only made to the Jenkins code base, so it won’t work with Hudson. Switch to Jenkins for the latest and most active continuous integration developments.
Can this work with GitHub as well ? Or do I have to use the GitHub Plugin
to do something similar if I have GitHub Enterprise internally at work ?
@mns
No, GitHub doesn’t allow you to directly touch post-receive hook. You should use the GitHub plugin for it, and it should work with GitHub Enterprise, too.
@eedri
I doubt if this works with multi SCM plugin. “Multi SCM” feature is something we really need to bring back into the core.
Unfortunately, I can verify this does not work with the MultiSCM plugin.
I’ve read there were plans to rewrite the git plugin to natively support multiple git repos, but given the last comment date was quite awhile ago, I’m wondering if this is happening or not. Or maybe I’m just out-of-the-loop here.
I am having trouble executing the build with this trigger in timely manner.
Namely, the trigger is executed, and the job is also found (and is valid / scheduled) but the build occurs about 10-20 minutes later.
I’ve done a bit of investigating and this is the flow and the information I can gather from logs:
* the project (e.g. jenkins-hooks-test) has a git(lab) repository, with the following web hook: http://ci-server/jenkins/git/notifyCommit?url=http://git.infobip.local/jenkins-hooks-test
* the project has received a push to the repository
* I can see that the hook fired the notifyCommit on jenkins, this is the trigger log:
01.10.2012. 14:46:46 hudson.triggers.SCMTrigger
FINE: Scheduling a polling for hudson.maven.MavenModuleSet@115be90[jenkins-hooks-test]
* the build starts 7 minutes later, the polling log:
Started on 01.10.2012. 14:53:00
Using strategy: Default
[poll] Last Build : #22
[poll] Last Built Revision: Revision 74a83490028ebf3b15af36a90d443f98a5e8262c (origin/master)
Fetching changes from the remote Git repositories
Fetching upstream changes from git@git.infobip.local:jenkins-hooks-test.git
Polling for changes in
Done. Took 32 sec
Changes found
So, from the chain of events point of view, this all OK since a change in the repository fires the build/polling trigger. The only problem I have is the delay. I’ve tested this when Jenkins is loaded with jobs but also when it’s building nothing.
The project is configured to Poll SCM @yearly and Build whenever a SNAPSHOT dependency is built. No other build triggers are present.
The main Jenkins configuration has the Quiet period set to 0.
I am using Jenkins 1.483 and git plugin 1.1.24.
Jenkins is configured with about 260 projects, of which most is currently using scheduled polling (every minute usually). I counted these by viewing all projects on the dashboard since the following snippet returns 1252 projects:
println Hudson.getInstance().getAllItems(AbstractProject.class).size()
(can’t figure out why the numbers don’t match)
I’ve also tried executing the trigger with this url: git@git.infobip.local:jenkins-hooks-test.git, and I get the same behaviour.
If used from curl, the response is:
“Scheduled polling of jenkins-hooks-test”
which notes that the build/polling trigger performed OK
(from what I can see in plugin code).
The only thing that is suspicious is probably the isStarving() on the trigger.
I’ve used the below script to get the information and it is in fact starving for 5 minutes (even more). I’ve also used the same script to run the trigger manually, with the same effect.
import hudson.plugins.git.*
import hudson.triggers.*
for (AbstractProject project : Hudson.getInstance().getAllItems(AbstractProject.class)) {
if(project.name.equals(“jenkins-hooks-test”)) {
println “project ${project.name}”
SCMTrigger trigger = project.getTrigger(SCMTrigger.class)
//println “executing the build trigger on the project”
//trigger.run()
println “the trigger is starving: ${trigger.descriptor.queue.isStarving(5 * 60 * 1000)}”
}
}
Any help appreciated!
P.S. If there is a smarter way to use gitlab with this plugin, I’d be very happy to give a hand…
P.P.S. the following starts the build instantly…
Maybe this is OK, since we are in fact “notifying of change to be built” ?
import hudson.plugins.git.*
import hudson.triggers.*
import hudson.model.*
for (AbstractProject project : Hudson.getInstance().getAllItems(AbstractProject.class)) {
if(project.name.equals(“jenkins-hooks-test”)) {
println “building project ${project.name}”
project.scheduleBuild(new Cause.RemoteCause(“gitlab”, “changes in the build”))
}
}
@Vanja Radovanović
I’ve created a gitlab hook plugin that offers both notify commit and build now options. This is not a direct solution but it works for us. You can find details @ https://wiki.jenkins-ci.org/display/JENKINS/Gitlab+Hook+Plugin
Hi. Can You tell me, please, where I need to add this command in jenkins?
How do I get this to work with all users of the repository? Git hooks appear to be user specific and are not saved in the repository.
I could not get the correct refs, but this worked for me:
for ref in $@; do
IFS=’/’ read -ra REF <<< "${ref}"
branch="${REF[2]}"
branch_list=${branch_list}${delim}${branch}
delim=","
done
I think its because we still have a old git version perhaps.
I have the setup working for git branches. Any new change triggers the builds configured.
@kohsuke
How do I make it work for any new git tag created manually? (As soon as a new tag is created, the hook causes the builds configured to trigger).
Hi, Kohsuke wrote “This will scan all the jobs that’s configured to check out the specified URL, and if they are also configured with polling, it’ll immediately trigger the polling (and if that finds a change worth a build, a build will be triggered in turn.) “.
When you say if they also configured with the polling, I guess you meant the Jenkin’s job, right?
If I don’t configure the Jenkin’s job to poll, then when does it trigger the build? I waited for about 23 mins still it didn’t trigger the build.
How can this work if the repository is placed on AWS and jenkins is running inside a firewall?
Either this plugin or Jenkins has a bug where specifying the git url of the form: `ssh://foo@bar.lan.local:7999/baz.git` in `http://yourserver/jenkins/git/notifyCommit?url=` above will give NullPointerException in Jenkins. This tool has too many bugs to be taken seriously (http://issues.jenkins-ci.org/secure/IssueNavigator.jspa?mode=hide&reset=true&jqlQuery=project+%3D+JENKINS+AND+status+in+%28Open%2C+%22In+Progress%22%2C+Reopened%29+AND+component+%3D+%27git%27)
Hi
I’d love to see this more clearly documented and discoverable. Right now it’s not obvious that you can use this instead of using a build that has a trigger URL and launches downstream builds when it’s triggered.
Forcing early polling, like this hook URL does, is much better, because it’ll schedule builds for any changes made since the last check. It can handle jobs that monitor multiple branches/tags. It won’t build a job that hasn’t changed. It’s just better.
Perhaps it should be referred to in the “?” entry for “Poll SCM” in the project or matrix plugins, alongside the existing “Configure Jenkins to poll changes in SCM. ” text?
… though on second thoughts, this looks a bit broken. When invoked it claimed to trigger all four jobs that depend on the repo in question, but in fact it only triggered one (the non-matrix job), not any of the three matrix jobs.
Does this work with git urls of the form git@bitbucket.org:/.git? My Jenkins job is set up with a git repository url like the one above. The build works fine. However when I try testing the hook url as above, I receive an error message:
Scheduled polling of
No Git consumers using SCM API plugin for: git@bitbucket.org:/.git
This seems quite odd given that my jenkins job name is not the same as my project name. So it is ending up at the right location – and then says it can’t figure out the location. ???
Grrrr, html made my substitutions disappear.
Urls of the format git@bitbucket.org:{group}/{project}.git?
and the error message is:
Scheduled polling of {jenkins job name}
No Git consumers using SCM API plugin for: git@bitbucket.org:{group}/{project}.git?
Hi Kohsuke,
can you please clarify how I can trigger jenkins job in case there is “new tag” added on a branch. My development team follows approach of tag based release and they don’t believe that each commit to a branch needs to be tested complete.
Please clarify
Thanks & Regards,
Vikram
@kohsuke
How do I do this using github?
Git hook notification method is only possible if your Jenkins is available on the internet with inbound access from GIT servers. If you are hosting jenkins privately inside your LAN with no inbound access from the internet then polling is your only option.
So polling can never die. The article is misleading.
Where i should execute below command on jenkins?
curl http://yourserver/jenkins/git/notifyCommit?url=
Hi Sir,
In my project
We will receive commit in GIT and latest git commit need to deploy in testing server(linux) automatically.
Please suggest where to mention linux testing server details and how to implement that.
Eg: In GIT received new commit for
a) \application\controllers\Common.php
b) \application\views\login.php
same need to deploy in linux server
JENKINS need to be deploy in same linux server? or anywhere
Hey,
I followed this blog and got my jenkins build to run on git commits, which is awesome! I was wondering if it is possible to run a jenkins build when a git branch is marked as release? The Build on GitSCM method of using git webhooks fails because jenkins looks specifically for a x-github-event of push, and not the release event.
I can get the webhook to push on release, but can’t get jenkins to build based off of that event.
Is there any way to build on branch “Release”?
Atleast I havent found any easy alternatives for running a certain build only once at particular time of the day only if there are changes.