Polling must die: triggering Jenkins builds from a git hook

December 1st, 2011

As I keep saying, polling a repository from Jenkins is inefficient; it adds delay on the order of minutes before a build starts after a commit is pushed, and it adds additional loads. It is much better instead to do push-notification from the repository. In this post, I’m going to explain how to do this for Git, which brings this on par with Subversion.

The previous best practice of doing this is best summarized in this blog post. While this works, this is less than ideal. A part of the problem is that this requires hard coding of job names inside the repository hooks, making it hard to keep them up-to-date. Another problem is that if your job only cares about one branch in a busy repository, you don’t want a new build to be triggered. Finally, the last problem is that you need some extra work for secured Jenkins.

With the latest Git plugin 1.1.14 (that I just release now), you can now do this more easily by simply executing the following command:

curl http://yourserver/jenkins/git/notifyCommit?url=<URL of the Git repository>

This will scan all the jobs that’s configured to check out the specified URL, and if they are also configured with polling, it’ll immediately trigger the polling (and if that finds a change worth a build, a build will be triggered in turn.) This allows a script to remain the same when jobs come and go in Jenkins. Or if you have multiple repositories under a single repository host application (such as Gitosis), you can share a single post-receive hook script with all the repositories. Finally, this URL doesn’t require authentication even for secured Jenkins, because the server doesn’t directly use anything that the client is sending. It runs polling to verify that there is a change, before it actually starts a build.

One more final note — in Git, unlike Subversion, a repository does not have its own identity, and a single repository sometimes have multiple URLs to access it. In such a case, simply execute the curl commands multiple times with all the different URLs.

That’s it. I hope this will help reduce the use of polling and have more people switch to push notifications. It really is addictive to see the build start in a split second after you push a change.

jenkins , ,

Introducing template plugin

November 15th, 2011

The Nectar 11.10 release that we just made has the template plugin, which I think is one of the very useful plugins that we added to Nectar that’s not available in open-source Jenkins. So I wanted to talk about it a bit.

Many serious Jenkins users have had this common problem that they’ve got a lot of jobs that look similar. JENKINS-3157, which is currently the most voted issue in JIRA, speaks to that same pain point. The template plugin is my answer to this problem.

What this plugin does, is to allow you to define a “model”. A model is an abstract concept that fits your particular problem domain. You do this through defining a set of attributes, kind of like how you define a class with a set of properties in Java, or how you define a table with a set of columns in database. You then define a transformation, which converts this model into something that Jenkins understands.

Creating a template

This is all rather abstract, so let’s look at a concrete example. We are going to define a hello world builder template, which lets you enforce the standard way to greet someone. You can then use this template to greet a lot of people in many jobs.

Installing this plugin, you get the “templates” menu on the left. You can click that and create a template:

You create a builder template from the same faimilar wizard you use to create a job:

Then here is the main meat.

There are two main things to configure here: attributes and a transformer.

When you define a template, you first ask yourself “what do I want my users to enter when they use my template?” The answer to that question becomes attributes. In this hello world builder, we want the user to configure who we are saying hello to, so we define one attribute named “target” for this purpose. The user should see the single text field for this, so we choose the type accordingly. The display name and inline help are self-explanatory. They control what the user will see when they are editing their free-style projects to use our new builder.

The second question you should ask when you define a template is “how does it execute?” (or more generally “how does it map to the terms Jenkins understands?”) The answer to that question becomes the transformer. In this tutorial, our builder will turn into a shell script that says hello (whereas your real template would probably turns into a shell script that gets some real work done, like building a component, running a test, etc.) So we’ll choose “generate a shell script to execute via Groovy”.

In the text area, we’ll fill the shell script that this build step is going to execute, but with expressions here and there of the form ${...} (because this is a Groovy template). ${target} refers to the actual value of the target attribute we defined above, and ${build.fullDisplayName} is a Groovy expression to access the getFullDisplayName() method (which returns the full name of the build) of the build object, which refers to the current build in progress. The ${build.fullDisplayName} needs to be quoted because this is going to look like test #1, and # gets interpreted by shell as a comment sign unless you quote it.

Using a template

Now that we have our first builder template, let’s create a free-style project that actually uses it. Go back to the Jenkins top page, and create a new free-style project. You’ll be taken to the configuration page. This part of Jenkins should be already familiar to you.

When you click “add build step”, you should see the newly created “Say Hello World” builder. You click it, and you see the say hello world builder added to your project. And you see the “target” attribute you defined in the configuration page.

I configured this to say hello to Kohsuke. If we save it and run it, lo and behold, you’ll see that it works.

[workspace] $ /bin/sh -xe /tmp/hudson2322666548865862105.sh
+ echo Hello to Kohsuke from test #1
Hello to Kohsuke from test #1
Finished: SUCCESS

This allows you as the build guy to create and enfroce a certain way your stuff gets built/tested, and your users get shielded from all the gory details of how the build/test actually get done. You can create a layer between Jenkins and your users, and make Jenkins talk in the language your users understand, instead of making them talk in the language Jenkins understands. This is one of the points of the template plugin.

Changing a template

Now, let’s change the definition of the template, and see how it affects the instances.

We’ll go back to the template definition by going back to the top page, clicking the “Templates” link in the left, and clicking the configuration icon on the right to the “Say Hello World” template.

Instead of saying hello, now we make it say good evening.Click “save” to save this new definition:

Now, when you update the template definition, all the use of this template automatically reflects the change you made. So without revisiting the configuration page of the freestyle job, let’s simply schedule another build and see its console output:

[workspace] $ /bin/sh -xe /tmp/hudson2322666548865862105.sh
+ echo Good evening to Kohsuke from test #2
Good evening to Kohsuke from test #2
Finished: SUCCESS

This is another point of the template plugin — when you change the definition, all the uses of this template get updated right away. So if you need to tweak how those high-level abstract concepts map to the lower level command sequences, you only need to edit it once and save it.

Try it out for yourself

If you like what you’ve read so far (I hope you do), please play with this. You can either download the whole Nectar and play it in a separate sandbox, or you can enable the secondary update center to install these plugins on your OSS Jenkins deployments.

And there’s a lot more to the template plugin, which I hope to write about in a few days.

jenkins , , ,

Quiz answer: memory leak in Java

November 4th, 2011

I posted a little quiz yesterday, and here is the answer.

The short answer is that InputStream needs to be closed. It’s easy to see why if it’s FileInputStream because you know the file handle needs to be released. But in this case, it’s just ByteArrayInputStream. We can just let GC recycle all the memory, right?

Turns out GZIPInputStream (or more precisely Deflater that it uses internally) uses native zlib code to perform decompression, so it’s actually occupying more memory (about 32K-64K depending on the compression level, I believe) on the native side, while its Java heap footprint is small. So if you allocate enough of those, you can end up eating a lot of native memory, while Java heap is still mostly idle. Even though those GZipInputStreams are no longer referenced, it just doesn’t create enough heap pressure to cause the GC to run.

And eventually you eat up all the native memory, and zlib’s malloc fails, and you get OutOfMemoryError (or your system starts to swap like crazy and your system effectively becomes unusable first.)

The other interesting thing to note is that -XX:HeapDumpOnOutOfMemoryError doesn’t do anything in this case. I read the JVM source code and I learned that heap dump only happens when OOME is caused during 3 or 4 specific memory allocation operations, like allocating a Java object, array, GC saturation, and a few other things. There are many other code passes in JVM that reports OOME, like this zlib malloc failure, that doesn’t trigger heap dump. There’s no question HeapDumpOnOutOfMemoryError is useful, but just beware that in some cases it doesn’t get created.

I knew that GZipInputStream is using native code internally, but I didn’t think about it too much when I was putting this original code together. Humans can’t think about all the transitive object graph and its implications.

The other lesson is that now I know why ps sometimes report such a big memory footprint for JVM while jmap reports only a modest usage. The difference is native memory outside Java heap, although unfortunately I don’t think there’s any easy way to check what’s eating the native memory.

My colleague and friend Paul Sandoz pointed out that if GZipInputStream was nice enough to free them up at EOF, it would have saved a lot of hassle, and I think he’s right — one still needs to consider the case where IOException causes the processing to abort before hitting EOF, but it would have helped, because those abnormal cases would be rare. I mean, there’s no harm in doing so, and anything that makes the library more robust in the face of abuse is a good thing, especially when the failure mode is this cryptic.

Uncategorized ,

Quiz time: memory leak in Java

November 3rd, 2011

Today I had an interesting debugging exercise, and I felt like I learned a new lesson that’s worth sharing with the rest of the world.

I had the following code, which takes a small-ish byte array and deserializes it into an object (let’s say someNotTooBigData is something like new byte[]{1,5,4, ... some data... }.) Seems innocent enough, no?

voidObject foo() {
	byte[] buf = someNotTooBigData();
	return new ObjectInputStream(new GZIPInputStream(
	    new ByteArrayInputStream(buf))).readObject();
}

But when this is executed frequently enough, like while(true) { foo(); }, it creates OutOfMemoryError. Can you tell why? I’ll post the answer tomorrow.

Uncategorized

Jenkins Community Survey

November 3rd, 2011

There’s currently a survey running to get a better sense of our use base. Those inputs help us steer the effort wisely, so we appreciate your taking time to fill it in. The result would be more useful if larger number of people participate, so feel free to encourage others to fill it in as well.

(The interesting thing about an open-source project is that you don’t get to tell other people what to work on — you can only try to influence them, and data like this is very useful.)

In the same spirit of the fundraising drive in NPR, CloudBees is throwing in a “thank you gift” of AppleTV as an added incentive. See their blog post for more details.

jenkins ,

Jenkins HackCamp at Tokyo

October 14th, 2011

Right after JavaOne, I traveled to Tokyo, and attended a 3-day hack camp. The Jenkins community had done several 1-day hackathon around the world, but this is the first that spanned across multiple days.

14 people came to a small traditional hotel in this small sea-side city of Ito, a 90 minutes train ride away from Tokyo city center, complete with a spa. Japanese people have a special place in their hearts for a spa, so we had to have one!

We stayed there for close to 48 hours, and the hacking was literally around the clock! I was still somewhat jet-lagged, so I went to sleep earlier, but when I woke up next morning around 6am, some people were still hacking code!

In between the hacking, we ate, drunk, talked, and took a bath. Some of us played board games that one of us brought (there seems to be a non-trivial intersection between the board game fans and open-source hackers.) There were a wide variety of activities, ranging from Arduino hardware hacking for XFD to automatic slave registration based on Android devices connected to USB, a bunch of automatic tool installers to classloader enhancements. Those results should be made online once people recovered from the trip.

One lesson for me for the next occasion is to plan for more ice-breaker events (which involves more drinking!) Maybe asking everyone to do a 5-10 minutes mini-talk about what they do and who they are.

jenkins , ,

My take on Jenkins User Conference

October 13th, 2011

Last week, we had our first ever Jenkins User Conference. More than 30 talks have been proposed (of which we were only able to accomodate 10 or so), 7 companies had helped pay for the event, and more than 250 people attended (out of 400 people registered.) People came from all over the world, from Japan, Korea, Australia, Europe, and so on. One of them even told me that he primarily came from Sweden for JUC, and being able to go to JavaOne was a nice extra. That’s saying something!

For me, to see the project I started 7 years ago to come this far, it was a rather special day. I was able to talk to many people during the day, most of the talks were quite good, and I enjoyed every bit of it. I hope other people felt the same way.

I’ve done the keynote that discusses the current state of the Jenkins project, and the progress we’ve made in the last 8 months since the divorce drama. The slides can be seen below, and I belive the recording will be posted online later.

Thank you for everyone who came, and thank you to those who made this event possible (especially Alyssa, Lisa, and Heidi who did the real hard work of organizing the event and taking care of the logistics.)
There was interest in doing another one in Europe. I certainly hope it would happen!

jenkins , ,

Ken Cavanaugh had passed away

August 25th, 2011

I’ve just learned that Ken Cavanaugh had passed away. He was my colleague back in Sun, and we had worked on a few small projects together.

When I joined Sun, he was already THE CORBA guy AFAIK, and when I left Sun, AFAIK he was still THE CORBA guy. And I was at Sun for, like 8 years. Not many people have a passion that lasts that long for any given field of technology, and that left me a rather strong impression. I always hoped that I could be like that when I get to his age.

Certain people emits an aura of confidence/reassurance. You can tell right away that he knows what he’s doing/saying when he does/says something. Ken was one of such people for me. He thus obviously commanded the respect he deserves, and you can see it in the guest posts that his colleagues left on his website that that’s not just me. I can really only use English well enough for dry technical matters, so I can’t really describe the feeling very well. I’m just very sorry to hear the news.

Uncategorized

My epic battle with findbugs-maven-plugin and how I utterly lost

August 23rd, 2011

It started quite innoucently. I was looking at this thread in Jenkins dev list and thought it’d be a good idea to get some critical findbugs errors to fail a build. My goal was simple, I want to run some high-priority findbugs checkers during the build, and if they report any error, I want the build to fail. I wanted this to be in a profile, so that I don’t need to wait for FindBugs to finish if I just want to build.

Should be simple enough, you’d think. Nope.

I’ve spent the entire afternoon getting this going. There were several issues in the plugin and relevant places that blocked my progress. In the hope that other people won’t suffer the same loss, here are those:

  • Maven 2.x and Maven 3.x site plugins are totally incompatible, breaking the setup that used to work. AFAIK there’s still no reasonable approach to enable your project to build both with Maven 2 and Maven 3, when it comes to site stuff, and that pretty much includes all the code analysis plugins. Maven 3 breaks backward compatibility with the site configuration of Maven 2, so the POM setup that used to work will no longer work with Maven 3. (So if someone tells you that Maven 3 is compatible with Maven2, don’t let them fool you.) It silently ignores all you have in and does nothing. So you’ll have to move the reporting configuration into a new location. This used to make it impossible to share the site configuration between Maven2 and Maven3, but I was told that the latest Maven site plugin version 3.0 no longer has this problem.

  • Findbugs plugin documentation seems to offer two mojos to generate reports — findbugs:check and findbugs:findbugs. But the check mojo actually isn’t capable of generating any reports. Two dozen or so configuration options to tweak the report generation you see in the doc are totally bogus. They are unused and ignored. (correction 8/24: what I missed is that the check plugin designates findbugs:findbugs as a pre-requisite.)

  • Some people tell you that you can invoke mvn findbugs:findbugs directly to generate report, but this is rather problematic if you actually try it. Firstly, it will generate XML but not HTML, so it’s useless for human beings. It does tell you how many bugs it found, but it doesn’t tell anything that actually points you to where the offending code is. One is supposed to be able to work around that by running the findbugs:gui mojo, but AFAICT this mojo is utterly broken. Secondly, if you invoke findbugs:findbugs mojo directly, it doesn’t pick up the same configuration that it uses during the site generation (one picks up build/plugins/plugin, the other looks at reporting/plugins/plugin). Again, AFAIK there’s no way to have those two modes of invocation use the same configuration.

  • You need to make sure that Maven at least compiled your source code before running site. The FindBugs mojo will happily skip itself if there’s no class files to work on, and unless you are smart enough to figure out what the mysterious “canGenerate=false” line means, you’ll waste your time trying to figure out why the mojo isn’t working, like I did.

  • Remember my use case of making the build fail in case of serious FindBugs issues? Documentation might make you believe that findbugs:check mojo is able to do this, but there are two large pit-falls. One is what I’ve already described, namely that this doesn’t actually run FindBugs, and instead it expects that you’ve already run it. The other is that if it doesn’t find any trace of FindBugs running, it happily skips itself. The consequence is that mvn clean install will always complete successfully, even if your code has FindBugs violations. I still haven’t figured out how to make this whole thing work. As I mentioned, findbugs report generation itself requires that the source code be compiled, so I guess you’d have to invoke Maven like mvn clean compile site install or something. This is just ridiculous.

  • In FindBugs, you can specify what rules you want to enforce and what rules you want to ignore. You describe this in a filter file. In a multi-module project, it tends to be more convenient to have just one filter file that all your modules use, rather than having many similar filter definitions. But this seemingly typical use case just doesn’t work with the Maven FindBugs plugin, because the path you specify in the filter file configuration is always interpreted relative to the current Maven module, and there seems to be no way to have it point to the base directory of the project (the ${project.basedir} macro also expands to the current module’s base directory, which is useless.) The documentation does talk about this and gives you a work around. As a Maven plugin developer myself, I understand where they are coming from, but as a Git user, the assumption that requires such a cumbersome workaround (of being able to check out and build modules individually, like you can in Subversion) is unnecessary, yet I still have to pay all the price. This doesn’t make sense.

I’m sorry to say this, but this is a disaster. Integrating FindBugs in Ant project, generating HTML report, and failing a build in case of significant error is fairly straight-forward, and takes maybe 10 or 20 lines at max. But here in Maven, it takes more lines in your POM, not to mention one whole Maven module just for the filter file, plus all these pitfalls. And it still doesn’t attain my original goal of making critical FindBugs issues fail the build.

Experiences like this made me really want to switch to Gradle, but alas, it’s no longer my call alone to make changes like that. So for the time being, I think I’m going back to my good trusted Maven antrun extended plugin. At least it works. And Stephen, this is why Ant fragment is actually more maintainable than the magical combination of Maven hacks.

Uncategorized ,

Calling for your participation in Jenkins User Conference

August 22nd, 2011

By now, you are surely aware of the Jenkins User Conference (JUC) that will be held the Sunday before JavaOne – October 2 – at the Marines’ Memorial Hotel in San Francisco, starting 9:00am PDT. This is a major milestone for the Jenkins Community – our first ever User Conference!

The Jenkins community needs to participate in JUC – certainly as an attendee, but also in supporting the organization and promotion of the Conference. Several vendors are helping to organize this first conference, but I think it’d be good for everyone if this is a community-driven event, and for that the Jenkins Community must play a major role, too.

To that end, I’d like to ask everyone to do the following to support JUC:

  • Register to attend. After all, it’s a free event with lots of useful contents!
  • Speak. People often incorrectly think that they need to be a project insider to be “qualified”, but that’s not the case. Lots of users want to hear about how other fellow users are using the software. So please tell us your showing off how you use Jenkins, clever ways you combine various pieces, technical/organizational challenges you faced, an so on. There are some great suggestions for potential topics on the JUC web page.
  • Recruit. If you aren’t comfortable speaking, please recruit someone else you think has an interesting Jenkins-related topic to present to the Community.
  • Promote the Conference. Even if you can’t come, please Tweet about it, re-Tweet posts you see from the @jenkinsconf Twitter account to your followers, email your friends, post to Facebook, and Like the JUC posts already there. If you paste in the JUC web page URL to your Facebook post, it will automatically pull in the cool Jenkins image we created for the conference. All of these things are easy to do and they really make a difference in getting the word out!
  • Sponsor the Conference. Sponsors are actively being recruited. One of the ways companies can give back to the Jenkins project that they’ve benefited from is by sponsoring JUC. Helping the project flourish protects your company’s investment in the use of Jenkins. All sponsorship levels also defray the costs of the Conference, thus keeping it free for attendees. Maybe your company would consider sponsoring? Hit up your favorite vendor to do so, too!

Registrations are coming in daily. So far, we have about 80 attendees, so please keep them coming. If you haven’t registered, register soon before the seats fill up. Making the event successful opens up many interesting options (like organizing other events in the future, etc), and that way it benefits the community as a whole, not to mention continuing to increase the Community’s knowledge and making Jenkins an even better platform in the process.

I look forward to seeing you there – as an attendee, a speaker, and/or a sponsor! We’ll have some more exciting things to announce in the coming weeks, so stay tuned!

jenkins , ,