Jenkins User Conference Israel

I’ll be visiting Israel this week, for Jenkins User Conference in Herzliya. I think this is my 3rd time visiting there, and I always enjoyed my visit and people over there (it really is an interesting place in many ways that everyone should visit once!) If you live in Israel, you can still register for the event!

You can see the planned talks, but there’ll be a lot of good “show and tell” talks where seasoned users share their experiences and setups to others, as well as more “emerging technologies” kind of talks, where people can get the idea of what’s cooking.

I’m planning to do a talk that contains a little bit of both; a bit about features that I developed for Jenkins that I use myself (some with my CloudBees hat on), then more about what the community has been working on in the core lately, and what I’d like to work on in near future.

It never ceases to amaze me that this piece of software called Jenkins has grown so much that we have 100s of people using it everywhere we go. Looking forward to meeting you all, and don’t forget to demand Jenkins stickers from me!

Push changes directly into BuildHive, and never run tests again!

On top of pull requests auto-build, BuildHive now allows you to push changes directly in via ssh. I call this feature “validated merge”.

If you are an active developer of a repository, chances are that you don’t use pull requests to send in changes. You probably just push changes directly into your repository instead. But one of the common problems of doing this is that if your change breaks the build, you are going to discover it too late. With validated merge, this problem will never happen again, and this is how it works.

First, you add the buildhive user as a collaborator to your repository, so that it can push when the build is done. I didn’t feel comfortable doing this automatically, so you need to do it explicitly.

Second, login to BuildHive (just once is suffice, so that it learns your public keys you registered with GitHub), head to the job page in BuildHive (like this), and click “Git repository for validated merge” link from the left. Click the copy button to copy the “git remote add” command, paste that in your shell, and execute that on your local repository. This will add BuildHive as a remote repository.

Preparation is now complete. Go modify code like you normally do by creating commits, and when you are ready to push, push your changes to “jenkins” instead of pushing to “origin”. If you are on a branch, replace “master” with the branch name you are working on:

$ git push jenkins master

BuildHive will check out the current tip of the specified branch in GitHub, merge that with the changes you just submitted, then do the build and run tests. If the merge result successfully completes the build and tests, the result will be pushed to GitHub by BuildHive.

If you are unlucky and your changes weren’t as good as you thought, you’ll learn that it didn’t pass the builds. You then stash your work, come back to the broken changes, and rework those. You probably don’t remember which commit you pushed to BuildHive, so BuildHive provides a tag you can fetch into your workspace to get to it.

$ git stash
$ git fetch -n ssh:// tag changes/45
$ git checkout changes/45
$ … make edits ...

You can add more commits to correct the problem, or you can even amend your commits — since your changes didn’t land in the repository, amending, rebasing, and so on is just fine. You can then re-push your changes and then go back to what you were working on before interrupted by a failure.

$ git push jenkins HEAD:master
$ git checkout master
$ git stash pop
… resume the work you were doing ...

If you’ve already headed home, your colleague can do this workflow for you.

Your time is precious. Don’t waste it by watching your laptop to complete tests. Instead, let BuildHive do it for you!

By the way, this is a feature already available in Jenkins Enterprise by CloudBees, so if you like the workflow but your source code isn’t in a public GitHub repository, you can do the same thing with any Git repository inside your firewall.

Jenkins Pry Plugin

I released Jenkins Pry plugin 1.1 today. This plugin adds the “pry” command to Jenkins CLI so that you can use pry to introspect a running Jenkins instance via pry.

Because this builds on top of Jenkins CLI, you can remotely connect from your laptop to your Jenkins master, safe with SSH public key authentication and transport encryption.

And because this builds on top of pry, it lets you interactively explore a rich object graph of Jenkins, doing bulk operations, checking the state of the system, etc.

I’ve done enough integration work so that the readline support in pry works as expected. So you can enjoy the code completion and command history support in pry, even though it’s running remotely. Nifty, isn’t it?

I hope this is useful for Jenkins admins who are more familiar with Ruby than Groovy (for which we had this “groovysh” CLI command for the longest time), but my real goal is to use this kind of remote Ruby code execution mechanism to let people run tests quickly when they develop Jenkins plugins in Ruby. This morning I’ve used this to verify that I can embed Cucumber tests. So stay tuned for more on that front!

The Butler’s Service: Promotion for Jenkins User Conference in Paris

We have just increased our enrollment capacity for the Jenkins User Conference (JUC) Paris, to be held on April 17th. The enthusiastic response to our first-ever Paris JUC has been terrific – and we want to get everyone there! The learning, networking and connecting that occurs within the Jenkins community at JUC is great to see. I saw it in spades last fall in San Francisco and it was terrific. I want every Jenkins user who is able to experience JUC to do so.

Since Mr. Jenkins, our iconic butler, and I are traveling to every JUC conference this year – all six of them – we have worked up a little scheme, with the folks from our sponsor CloudBees, to get YOU there, too.

Here is the deal. Jenkins and I are offering a special Butler’s Service promotion for JUC Paris. Ticket prices recently increased to the full conference price of €206, from a previously discounted rate of €104. The registration fee is needed to cover the cost of the conference, but we realize this can get in the way of people trying to attend. As a compromise, on this Thursday and Friday, 5-6 April, we will reduce the ticket price to the lower advance registration special pricing that was in effect. So on Thursday and Friday, you can still register for JUC Paris at the lower €104 price. (For anyone who paid the full conference price, we will refund the difference.)

Let Your Friends and Colleagues Know About This Special Offer!

To experience what JUC is like, watch the highlights video from our San Francisco conference. It will give you a feel for the quality of our speakers, the learning – and, yes, the fun that went on!

We have a lot to share with you on April 17. In addition to all of the great sessions we are offering, you’ll get to see many developers of the plugins you’ve been using (some of whom even I haven’t met before in person!), and there will be some exciting updates and other news to share about our favorite continuous integration platform. You will want to be there to hear and see it all, first hand.

I hope to see you on April 17 in Paris – be sure to sign up by end-of-day, Friday, and take advantage of this great deal served up by the Butler!

Come join us on “Selenium, Jenkins, Robots, Oh My!” tomorrow

I’ll be speaking tomorrow at San Francisco Selenium Meetup about Jenkins & Selenium — mainly recent improvements in the Selenium plugin, as well as several other new plugins relevant in the combination of Jenkins and Selenium, complete with a demo. I’ve got a couple of pet-peeves against the Selenium project, so I’m going to pitch them there to see the reaction, too.

I’ve told that Jason Huggins from Sauce Lab is going to pick up where he left off in the last Jenkins User Conference to talk about his robot, and Theo Cincotta from Eventbrite will give the case study of how Eventbrite uses Jenkins & Selenium together internally, so the whole thing should be a great mix of fun & useful topics, all packed in a Wednesday night from 6:30pm to 8:00pm, with beer and food.

The event is free, but you do need to RSVP, instead of the usual RSVP in the SF Selenium meetup group page at (which currently says 209 people coming, when the EventBrite RSVP page says the capacity is 100 people — so I need to check with the organizers to make sure they know what they are expecting…)

Attaching files to JUnit tests

Despite the fact that it is the de-facto standard of test reports in any programming languages (perhaps except .NET), JUnit test report format has a number of problems. One is that the format isn’t explicitly defined (and I’ll discuss this in a separate post), but another problem, which I’m going to dedicate this post for, is the lack of attachment support.

It is often very conveniet to be able to attach arbitrary files to a test report. Imagine Selenium tests capturing screenshots. Or JavaEE tests that deploy webapps, then you want to capture server log files. Or if you are doing UI automation testing, how about a video recording that your screen automation framework has produced?

Today, doing this mostly requires that you (as a test author) write some files somewhere, then print out that file name to stdout/stderr. This works for humans who are looking at the output, but not for CI servers like Jenkins. So I hereby propose a bit of convention to decorate this current practice, to make those files discoverable by Jenkins.

For this, I’ve improved the JUnit attachments plugin to recognize the following format. It has to occupy a whole line.

[[ATTACHMENT|<absolute file name>]]
[[ATTACHMENT|<absolute file name>|{ ... additional metadata in JSON ... }]]

The additional metadata isn’t currently used, but I intended it to describe what the attached file means. For example, if your test always attach a couple of log files, it’d be useful to describe which file is which, so that CI servers or test report tools can display them as such. Or metadata for human readable display name would be useful, as these attachment file names are often cryptic just to make them unique.

When you run these tests from within Jenkins, these files are then picked up and stored by Jenkins, and the test report page will include them as links.

Ideally, the test report format should be expanded to cover things like this, but unfortunately I think that’d require too much collaboration between too many people to the point that it’s unrealistic — if we are to do that, test frameworks like JUnit first needs to offer this as API methods for listeners, then the test drivers like Ant/Maven needs to be expanded to honor those when they produce test reports. Then finally we can improve the CI servers.

I’ve been patiently waiting for that to happen for long time, but it’s just not happening. So instead, I’m taking the matter into my own hands, and came up with this convention.

Convention like this is useful only if enough people uses it. So I hope you’ll like this. If you think this convention can be improved, please let me know.

Jenkins User Conferences 2012

As a result of the success of the Jenkins User Conference last year in San Francisco, this year CloudBees and other sponsors are planning 4 Jenkins User Conferences around the world. Yup, that’s right — four!

Making events like these successful would be a win for everyone, so I’d like to encourage you to …:

Register to attend

If you are in the greater Paris or NY City metro areas, sign up now! You will experience lots of great learning opportunities and further your Jenkins knowledge. And perhaps more importantly, you’ll get the chance to connect with other Jenkins users and learn about the latest developments in the Jenkins project. I always love watching the community interact – we all learn from each other and on so many occasions, I have seen a Jenkins user have an “Aha!” moment in conversation with another Jenkins user. There’s something magical about meeting face-to-facet that you just can’t replace any other way.

Dates for the JUC conferences in the first half of 2012 are:

  • Paris — April 17: Early Bird discounted registration (€50) ends 18 February

  • New York City — May 17: Early Bird discounted registration ($54) ends 25 February

  • Registration includes all sessions, lunch, a social/networking hour at the end of the day and a FREE Jenkins t-shirt (a hot commodity at the JUC last October)

  • Two more conferences will be scheduled later in the year in San Francisco(Sept) and Antwerp (Nov), and there’s a planning going on for one in Tokyo — stay tuned for dates.
Apply to speak

Share your Jenkins knowledge with the community. The quality of the speakers was the most highly praised aspect of JUC 2011. If you’ve been writing plugins, tell the community what those are and meet people who use your plugins, which is quite a rewarding experience. If you’ve been using Jenkins in interesting ways, tell the community about it, and through the feedback you’ll learn just as much as they would. Do it fast though — the deadline for Paris is February 24th and for New York is March 16th.

Sign up to sponsor

If you can, consider sponsoring the events. If you feel your organization has benefitted from Jenkins, now’s your chance to give back to the community… and at the same time get some great visibility for your company.


If you still need more convincing about JUC, watch the highlights video from last year’s conference. Looking forward to seeing you!

This week in Tokyo

I’m back to Tokyo this and next week, doing all sorts of Jenkins related (and other CloudBees related) activities.

First was the Developer Summit, a two-day developer conference. It covers wide range of topics from mobile to web, agile to industry designs. Yesterday I’ve done a lightening talk, trying to convince them the importance of communicating and sharing (be it code, blog, etc.) The subtext was to do so in English, ideally, so that the rest of the world can see all the great stuff that they are doing, which for me has been a constant struggle in the Jenkins community. During this lightning talk the moderator asked the audience how many of them already use Jenkins, and about 1/3 of the hands went up. Then he asked how many already knew Jenkin, and about another 1/3 went up. So I was quite encouraged.

Today in its 2nd day, I’m doing my high-level Jenkins talk, trying to spread the words and explaining why CI (especially in the context of abidance of computing power) isn’t a transient phenomena, but instead a profound shift in how we develop.

On Tuesday next week, we’ll be doing the 5th Tokyo Jenkins user meetup. Thanks to the usual suspects in the Japanese Jenkins community and a kind offer from Rakuten (a Japanese equivalent of Yahoo, would be the best way to describe this company, I think), this time we’ve got a huge venue, but we’ve already filled up all 110 seats. We have a great speaker line-up, folks from Rakuten and Mixi (a Japanese equivalent of Facebook) discussing how they deploy CI in those large organizations.

I’ve heard from many yesterday that they couldn’t get in because seats are full. In one of those days, I’d really love to bring Jenkins User Conference to Tokyo in some shape. We’ll have to do with a lot less CloudBees involvement (and that means I need to find someone else who can help organize the event, as those things are quite a bit of work), but I think it’d be worth it.

The day after that, I’ll be with Mamezou (whom CloudBees partner with to deliver training in Tokyo) and do a short introductory talk/demo on Jenkins. Whereas the user meetup is more focused on existing users and building relationship among them, this event is more for new users who have no prior experience on Jenkins. Tamagawa-san, who translated “Jenkins the Definitive Guide” to Japanese, would also come to give a presentation.

I’ll have my usual one day training on Thursday, then on Friday I’ll be at the cloud user group meet-up to present about the CloudBees platform.

I’ve brought 300 Jenkins stickers with me, but I’ve already handed out about 50 today, and I’m already regretting that I haven’t brought more. The same goes to business cards — I don’t know what it is but Japanese sure loves exchanging business cards!

In between those I’ve got some company visits and other engagements lined up, and if I get lucky I’ll be meeting&amp;drinking with some old friends. This weekend is likely spent preparing for talks and catching up with work during my Europe trip, but I’ve got the next weekend after all my commitments are done, so I should be able to do some unwinding, too.

All in all, it’s going to be a busy week. My apologies in advance for additional delay in e-mail responses, etc.

Writing programs that drive Jenkins

One of the interesting discussions during SCALE 10x was about using Jenkins as a piece of a bigger software. This person was interested in using Jenkins to run some business analysis operations, and wanted to have a separate interface for business oriented people. This is actually an emerging but common theme I hear from many users. Another company in San Jose actually built the workflow engine that uses Jenkins as a piece in a bigger application (aside from the actual build and test, this workflow involves reviews, approvals, etc.), and GitHub Janky can be classified as one such app, too.

This is something I always believed in — that every piece of software needs to be usable by a layer above. Or put another way, every software should be usable as a library.

So in this post I’m going to discuss various ways you can programmatically drive Jenkins.

Let’s start with the REST API of Jenkins. For most of the data Jenkins renders as HTML, you can access its XML version and JSON version (as well as a few other formats, like Python literal fragment.) You do this by adding /api to the page (see for example.) Those pages discusss other REST API where applicable. For example, you can POST to certain URL and it’ll create/update job definitions, etc.

If you are going to use REST API, you might find the auto-discovery for Jenkins useful. You can discover Jenkins on the local subnet via UDP broadcast, or DNS multi-cast. There’s also a distinctive HTTP header “Jenkins-Version” on the top page of Jenkins that allows your application to verify that it’s talking to a real Jenkins server, as well as an instance ID that allows you to identify Jenkins instanecs. These features allow you to build smarter applications.

For Jenkins protected by some authentication mechanism, you can use the user name + API key in the HTTP basic auth (and I want to add OAuth support here.)

REST API is great that it’s programming language agnostic. It is also convenient that neither the server nor the client has to trust each other. But those APIs are bound by the request/response oriented nature of the HTTP protocol.

Another great integration point for Jenkins is the CLI. This uses the same underlying technology that drives master/slave architecture, which enables your command line clients to be a lot more intelligent. For example, REST API exposes an URL that you can post to get a build started. But the equivalent command in CLI can have you block until the build is complete (and exit code indicates the status), or run the polling first and proceed to build only when the polling detects a change, or allow you perform a parameterized build with multiple file uploads very easily. For protected Jenkins, CLI supports SSH public key authentication to securely authenticate the client.

A slightly different version of the CLI is “Jenkins as SSH server”. Jenkins speaks the server side of the SSH protocol, and allow regular SSH clients to execute a subset of CLI commands. In this way, you don’t need any Java runtime installed on the client side to drive Jenkins.

These two integration APIs are often much easier to script than REST API.

Those APIs are available for non-privileged users, and they are great for small scale integrations. But for more sophisticated integration needs, we have additional APIs.

One is the REST API access to the Groovy console, which allows administrator users to run arbitrary Groovy script inside the Jenkins master JVM (and you can submit this script as POST payload, and get the response back as the HTTP response.) This allows you to tap into all the Jenkins object models. Unlike the REST API, in this way you can ship the computation, so in one round-trip you can do a lot. You can do the same with CLI, which also lets you access stdin/stdout of the CLI.

The other sophisticated integration API I wanted to talk about is the remoting API that does Java RPC (not to be confused with the remote API, which is the synonym for the REST API.) The remoting API is the underlying protocol that we use for master/slave communications, and it revolves around the notion of shipping a closure (and code associated with it) from one JVM to another, execute it, and get the result back. If your application runs elsewhere, you can establish the remoting API channel with Jenkins master, then prepare a Callable object. You can then have Jenkins master execute this closure, and the result is sent back to your JVM.

There’s an example of this available. You bootstrap this in the same way the CLI client talks to the master, then you “upgrade” the communication channel by activating the remote code download support (which requires the administrator privilege, for obvious reasons.)

The great thing about this is that your data structure is rich Java object model all the way, and you never have to translate your data to externalizable serialization data format like XML or JSON. This greatly simplifies your program.

I think this list covers all the major integration APIs that Jenkins offers. If you are building any interesting applications that uses Jenkins as a building block, please share your experience so that we can make it better!

Jenkins ChromeDriver plugin

I released Jenkins ChromeDriver plugin that auto-installs chromedriver on all slaves. Here is why this is useful.

ChromeDriver is needed to run WebDriver-based tests with Chrome. Unfortunately this is a native program, so in a heterogenous Jenkins, you need to make sure you have the right binary available on each of the slaves. While that’s not hard, it’s awfully tedious. This plugin fixes that by automating it.

This plugin would be a nice companion to the recently upgraded Selenium Grid plugin that automatically runs Selenium Grid on top of your Jenkins slaves. When you combined, you just install two plugins to the Jenkins master, and without touching slaves at all, you got the whole cluster instantly ready for Selenium2 WebDriver testing. How is that?