Kohsuke Kawaguchi

Today I’m going to talk about the upcoming feature in Jenkins 1.442 (to be released today), which allows you to finally install plugins without restarting Jenkins.

As I’ll discuss later, internally this is somewhat of an involving work. But for users, this is a very simple feature that doesn’t involve any learning curve. You’ll go to the plugin manager, click “Available” tab, then choose the plugins to install. Scroll all the way down, then you’ll see the “Install without restart” button as well as the “Download new and install after restart” button. The former is the result of this work, allowing you to start using the new plugins right away. The latter is the traditional behaviour, where new plugins take effect after the next restart.

I’m even thinking about removing the latter button after a while, once we gain enough confidence in the feature. I can’t think of any inherent reasons the latter is desirable over the former.

Click the button on the left, and the plugin gets downloaded, installed, and activated:

Whereas existing users would by now be “trained” to click the “restart Jenkins when done” checkbox, now you can just go back to the top page and start using the new features from this plugin right away. Yup, that’s all there is to it.

But how about upgrades?

Unfortunately, because of the architectual choice made in Jenkins, this same scheme wouldn’t allow us to upgrade existing plugins. See below.

But how about uninstallation?

Likewise, this same scheme wouldn’t really let us do the “uninstallation” in the normal sense of the word. But I think some fake of that might be possible — like disabling all the contributions from this plugin (which will prevent anyone from using builders/publishers/etc in newly created jobs), but leaving existing jobs with those features configured as-is, until the next reboot, at which point those features will be just forgotten.

Internal

Jenkins added the plugin support in 1.44, which is some 400 releases ago (man I’ve been doing this for long time!), and since that time Jenkins required a restart for new plugins to take effect. This is because we load plugins at start-up, doing all kinds of computation to build up the immutable data structure around all the plugins. Thus requiring a restart was partly out of my laziness, partly my preference to the immutable data structures, and partly because the computation was rather sequential. And I had bigger fishes to fry, like update centers, and things were left as it was.

Then later came the improvement to the boot up, which removed a long sequential code, and divided them up to the directed acyclic graph of smaller tasks, which we then execute in parallel. This was primarily to speed up the boot time, but this made it somewhat easier to do dynamic initialization as a side effect. Then further down the road, the introduction of Guice made it harder once again, as it prefers immutability, too. So the current implementation puts the new components into the child injector, which seems to make enough plugins happy.

Finally, why is upgrade hard? This is because Jenkins keeps instances of model objects for long time in memory (unlike, say, a typical database application, where those things are request scoped and thus much short-lived.) I like this for a number of reasons, such as more straight-forward object traversals, and the fact that builds take a long time anyway. But as with everything else in the software design, it is not without downsides, and one of them is that it makes dynamic reloading harder. This hopefully answers why just using OSGi alone does not fix this problem.

My current thinking is that “fixing” this via HA is probably the way to go. Or maybe fake uninstallation + dynamic installation might be enough for most plugins. We’ll see.

Conclusion

Hopefully you’ll like this feature. While this is an obvious simple feature for users, as you see internally I’ve done some hard work. If this makes it easier for you to try out new plugins, my time was well spent.

jenkins install, plugin

As I keep saying, polling a repository from Jenkins is inefficient; it adds delay on the order of minutes before a build starts after a commit is pushed, and it adds additional loads. It is much better instead to do push-notification from the repository. In this post, I’m going to explain how to do this for Git, which brings this on par with Subversion.

The previous best practice of doing this is best summarized in this blog post. While this works, this is less than ideal. A part of the problem is that this requires hard coding of job names inside the repository hooks, making it hard to keep them up-to-date. Another problem is that if your job only cares about one branch in a busy repository, you don’t want a new build to be triggered. Finally, the last problem is that you need some extra work for secured Jenkins.

With the latest Git plugin 1.1.14 (that I just release now), you can now do this more easily by simply executing the following command:

curl http://yourserver/jenkins/git/notifyCommit?url=<URL of the Git repository>

This will scan all the jobs that’s configured to check out the specified URL, and if they are also configured with polling, it’ll immediately trigger the polling (and if that finds a change worth a build, a build will be triggered in turn.) This allows a script to remain the same when jobs come and go in Jenkins. Or if you have multiple repositories under a single repository host application (such as Gitosis), you can share a single post-receive hook script with all the repositories. Finally, this URL doesn’t require authentication even for secured Jenkins, because the server doesn’t directly use anything that the client is sending. It runs polling to verify that there is a change, before it actually starts a build.

One more final note — in Git, unlike Subversion, a repository does not have its own identity, and a single repository sometimes have multiple URLs to access it. In such a case, simply execute the curl commands multiple times with all the different URLs.

That’s it. I hope this will help reduce the use of polling and have more people switch to push notifications. It really is addictive to see the build start in a split second after you push a change.

jenkins git, jenkins, polling

The Nectar 11.10 release that we just made has the template plugin, which I think is one of the very useful plugins that we added to Nectar that’s not available in open-source Jenkins. So I wanted to talk about it a bit.

Many serious Jenkins users have had this common problem that they’ve got a lot of jobs that look similar. JENKINS-3157, which is currently the most voted issue in JIRA, speaks to that same pain point. The template plugin is my answer to this problem.

What this plugin does, is to allow you to define a “model”. A model is an abstract concept that fits your particular problem domain. You do this through defining a set of attributes, kind of like how you define a class with a set of properties in Java, or how you define a table with a set of columns in database. You then define a transformation, which converts this model into something that Jenkins understands.

Creating a template

This is all rather abstract, so let’s look at a concrete example. We are going to define a hello world builder template, which lets you enforce the standard way to greet someone. You can then use this template to greet a lot of people in many jobs.

Installing this plugin, you get the “templates” menu on the left. You can click that and create a template:

You create a builder template from the same faimilar wizard you use to create a job:

Then here is the main meat.

There are two main things to configure here: attributes and a transformer.

When you define a template, you first ask yourself “what do I want my users to enter when they use my template?” The answer to that question becomes attributes. In this hello world builder, we want the user to configure who we are saying hello to, so we define one attribute named “target” for this purpose. The user should see the single text field for this, so we choose the type accordingly. The display name and inline help are self-explanatory. They control what the user will see when they are editing their free-style projects to use our new builder.

The second question you should ask when you define a template is “how does it execute?” (or more generally “how does it map to the terms Jenkins understands?”) The answer to that question becomes the transformer. In this tutorial, our builder will turn into a shell script that says hello (whereas your real template would probably turns into a shell script that gets some real work done, like building a component, running a test, etc.) So we’ll choose “generate a shell script to execute via Groovy”.

In the text area, we’ll fill the shell script that this build step is going to execute, but with expressions here and there of the form ${...} (because this is a Groovy template). ${target} refers to the actual value of the target attribute we defined above, and ${build.fullDisplayName} is a Groovy expression to access the getFullDisplayName() method (which returns the full name of the build) of the build object, which refers to the current build in progress. The ${build.fullDisplayName} needs to be quoted because this is going to look like test #1, and # gets interpreted by shell as a comment sign unless you quote it.

Using a template

Now that we have our first builder template, let’s create a free-style project that actually uses it. Go back to the Jenkins top page, and create a new free-style project. You’ll be taken to the configuration page. This part of Jenkins should be already familiar to you.

When you click “add build step”, you should see the newly created “Say Hello World” builder. You click it, and you see the say hello world builder added to your project. And you see the “target” attribute you defined in the configuration page.

I configured this to say hello to Kohsuke. If we save it and run it, lo and behold, you’ll see that it works.

[workspace] $ /bin/sh -xe /tmp/hudson2322666548865862105.sh
+ echo Hello to Kohsuke from test #1
Hello to Kohsuke from test #1
Finished: SUCCESS

This allows you as the build guy to create and enfroce a certain way your stuff gets built/tested, and your users get shielded from all the gory details of how the build/test actually get done. You can create a layer between Jenkins and your users, and make Jenkins talk in the language your users understand, instead of making them talk in the language Jenkins understands. This is one of the points of the template plugin.

Changing a template

Now, let’s change the definition of the template, and see how it affects the instances.

We’ll go back to the template definition by going back to the top page, clicking the “Templates” link in the left, and clicking the configuration icon on the right to the “Say Hello World” template.

Instead of saying hello, now we make it say good evening.Click “save” to save this new definition:

Now, when you update the template definition, all the use of this template automatically reflects the change you made. So without revisiting the configuration page of the freestyle job, let’s simply schedule another build and see its console output:

[workspace] $ /bin/sh -xe /tmp/hudson2322666548865862105.sh
+ echo Good evening to Kohsuke from test #2
Good evening to Kohsuke from test #2
Finished: SUCCESS

This is another point of the template plugin — when you change the definition, all the uses of this template get updated right away. So if you need to tweak how those high-level abstract concepts map to the lower level command sequences, you only need to edit it once and save it.

Try it out for yourself

If you like what you’ve read so far (I hope you do), please play with this. You can either download the whole Nectar and play it in a separate sandbox, or you can enable the secondary update center to install these plugins on your OSS Jenkins deployments.

And there’s a lot more to the template plugin, which I hope to write about in a few days.

jenkins cloudbees, jenkins, nectar, template

I posted a little quiz yesterday, and here is the answer.

The short answer is that InputStream needs to be closed. It’s easy to see why if it’s FileInputStream because you know the file handle needs to be released. But in this case, it’s just ByteArrayInputStream. We can just let GC recycle all the memory, right?

Turns out GZIPInputStream (or more precisely Deflater that it uses internally) uses native zlib code to perform decompression, so it’s actually occupying more memory (about 32K-64K depending on the compression level, I believe) on the native side, while its Java heap footprint is small. So if you allocate enough of those, you can end up eating a lot of native memory, while Java heap is still mostly idle. Even though those GZipInputStreams are no longer referenced, it just doesn’t create enough heap pressure to cause the GC to run.

And eventually you eat up all the native memory, and zlib’s malloc fails, and you get OutOfMemoryError (or your system starts to swap like crazy and your system effectively becomes unusable first.)

The other interesting thing to note is that -XX:HeapDumpOnOutOfMemoryError doesn’t do anything in this case. I read the JVM source code and I learned that heap dump only happens when OOME is caused during 3 or 4 specific memory allocation operations, like allocating a Java object, array, GC saturation, and a few other things. There are many other code passes in JVM that reports OOME, like this zlib malloc failure, that doesn’t trigger heap dump. There’s no question HeapDumpOnOutOfMemoryError is useful, but just beware that in some cases it doesn’t get created.

I knew that GZipInputStream is using native code internally, but I didn’t think about it too much when I was putting this original code together. Humans can’t think about all the transitive object graph and its implications.

The other lesson is that now I know why ps sometimes report such a big memory footprint for JVM while jmap reports only a modest usage. The difference is native memory outside Java heap, although unfortunately I don’t think there’s any easy way to check what’s eating the native memory.

My colleague and friend Paul Sandoz pointed out that if GZipInputStream was nice enough to free them up at EOF, it would have saved a lot of hassle, and I think he’s right — one still needs to consider the case where IOException causes the processing to abort before hitting EOF, but it would have helped, because those abnormal cases would be rare. I mean, there’s no harm in doing so, and anything that makes the library more robust in the face of abuse is a good thing, especially when the failure mode is this cryptic.

Uncategorized java, quiz

Today I had an interesting debugging exercise, and I felt like I learned a new lesson that’s worth sharing with the rest of the world.

I had the following code, which takes a small-ish byte array and deserializes it into an object (let’s say someNotTooBigData is something like new byte[]{1,5,4, ... some data... }.) Seems innocent enough, no?

voidObject foo() {
	byte[] buf = someNotTooBigData();
	return new ObjectInputStream(new GZIPInputStream(
	    new ByteArrayInputStream(buf))).readObject();
}

But when this is executed frequently enough, like while(true) { foo(); }, it creates OutOfMemoryError. Can you tell why? I’ll post the answer tomorrow.

Uncategorized java

There’s currently a survey running to get a better sense of our use base. Those inputs help us steer the effort wisely, so we appreciate your taking time to fill it in. The result would be more useful if larger number of people participate, so feel free to encourage others to fill it in as well.

(The interesting thing about an open-source project is that you don’t get to tell other people what to work on — you can only try to influence them, and data like this is very useful.)

In the same spirit of the fundraising drive in NPR, CloudBees is throwing in a “thank you gift” of AppleTV as an added incentive. See their blog post for more details.

jenkins jenkins, survey

Right after JavaOne, I traveled to Tokyo, and attended a 3-day hack camp. The Jenkins community had done several 1-day hackathon around the world, but this is the first that spanned across multiple days.

14 people came to a small traditional hotel in this small sea-side city of Ito, a 90 minutes train ride away from Tokyo city center, complete with a spa. Japanese people have a special place in their hearts for a spa, so we had to have one!

We stayed there for close to 48 hours, and the hacking was literally around the clock! I was still somewhat jet-lagged, so I went to sleep earlier, but when I woke up next morning around 6am, some people were still hacking code!

In between the hacking, we ate, drunk, talked, and took a bath. Some of us played board games that one of us brought (there seems to be a non-trivial intersection between the board game fans and open-source hackers.) There were a wide variety of activities, ranging from Arduino hardware hacking for XFD to automatic slave registration based on Android devices connected to USB, a bunch of automatic tool installers to classloader enhancements. Those results should be made online once people recovered from the trip.

One lesson for me for the next occasion is to plan for more ice-breaker events (which involves more drinking!) Maybe asking everyone to do a 5-10 minutes mini-talk about what they do and who they are.

jenkins jenkins, spa, tokyo

Last week, we had our first ever Jenkins User Conference. More than 30 talks have been proposed (of which we were only able to accomodate 10 or so), 7 companies had helped pay for the event, and more than 250 people attended (out of 400 people registered.) People came from all over the world, from Japan, Korea, Australia, Europe, and so on. One of them even told me that he primarily came from Sweden for JUC, and being able to go to JavaOne was a nice extra. That’s saying something!

For me, to see the project I started 7 years ago to come this far, it was a rather special day. I was able to talk to many people during the day, most of the talks were quite good, and I enjoyed every bit of it. I hope other people felt the same way.

I’ve done the keynote that discusses the current state of the Jenkins project, and the progress we’ve made in the last 8 months since the divorce drama. The slides can be seen below, and I belive the recording will be posted online later.

Thank you for everyone who came, and thank you to those who made this event possible (especially Alyssa, Lisa, and Heidi who did the real hard work of organizing the event and taking care of the logistics.)
There was interest in doing another one in Europe. I certainly hope it would happen!

jenkins javaone, jenkins, juc

I’ve just learned that Ken Cavanaugh had passed away. He was my colleague back in Sun, and we had worked on a few small projects together.

When I joined Sun, he was already THE CORBA guy AFAIK, and when I left Sun, AFAIK he was still THE CORBA guy. And I was at Sun for, like 8 years. Not many people have a passion that lasts that long for any given field of technology, and that left me a rather strong impression. I always hoped that I could be like that when I get to his age.

Certain people emits an aura of confidence/reassurance. You can tell right away that he knows what he’s doing/saying when he does/says something. Ken was one of such people for me. He thus obviously commanded the respect he deserves, and you can see it in the guest posts that his colleagues left on his website that that’s not just me. I can really only use English well enough for dry technical matters, so I can’t really describe the feeling very well. I’m just very sorry to hear the news.

Uncategorized tribute

It started quite innoucently. I was looking at this thread in Jenkins dev list and thought it’d be a good idea to get some critical findbugs errors to fail a build. My goal was simple, I want to run some high-priority findbugs checkers during the build, and if they report any error, I want the build to fail. I wanted this to be in a profile, so that I don’t need to wait for FindBugs to finish if I just want to build.

Should be simple enough, you’d think. Nope.

I’ve spent the entire afternoon getting this going. There were several issues in the plugin and relevant places that blocked my progress. In the hope that other people won’t suffer the same loss, here are those:

• Maven 2.x and Maven 3.x site plugins are totally incompatible, breaking the setup that used to work. AFAIK there’s still no reasonable approach to enable your project to build both with Maven 2 and Maven 3, when it comes to site stuff, and that pretty much includes all the code analysis plugins. Maven 3 breaks backward compatibility with the site configuration of Maven 2, so the POM setup that used to work will no longer work with Maven 3. (So if someone tells you that Maven 3 is compatible with Maven2, don’t let them fool you.) It silently ignores all you have in and does nothing. So you’ll have to move the reporting configuration into a new location. This used to make it impossible to share the site configuration between Maven2 and Maven3, but I was told that the latest Maven site plugin version 3.0 no longer has this problem.

• Findbugs plugin documentation seems to offer two mojos to generate reports — findbugs:check and findbugs:findbugs. But the check mojo actually isn’t capable of generating any reports. Two dozen or so configuration options to tweak the report generation you see in the doc are totally bogus. They are unused and ignored. (correction 8/24: what I missed is that the check plugin designates findbugs:findbugs as a pre-requisite.)

• Some people tell you that you can invoke mvn findbugs:findbugs directly to generate report, but this is rather problematic if you actually try it. Firstly, it will generate XML but not HTML, so it’s useless for human beings. It does tell you how many bugs it found, but it doesn’t tell anything that actually points you to where the offending code is. One is supposed to be able to work around that by running the findbugs:gui mojo, but AFAICT this mojo is utterly broken. Secondly, if you invoke findbugs:findbugs mojo directly, it doesn’t pick up the same configuration that it uses during the site generation (one picks up build/plugins/plugin, the other looks at reporting/plugins/plugin). Again, AFAIK there’s no way to have those two modes of invocation use the same configuration.

• You need to make sure that Maven at least compiled your source code before running site. The FindBugs mojo will happily skip itself if there’s no class files to work on, and unless you are smart enough to figure out what the mysterious “canGenerate=false” line means, you’ll waste your time trying to figure out why the mojo isn’t working, like I did.

• Remember my use case of making the build fail in case of serious FindBugs issues? Documentation might make you believe that findbugs:check mojo is able to do this, but there are two large pit-falls. One is what I’ve already described, namely that this doesn’t actually run FindBugs, and instead it expects that you’ve already run it. The other is that if it doesn’t find any trace of FindBugs running, it happily skips itself. The consequence is that mvn clean install will always complete successfully, even if your code has FindBugs violations. I still haven’t figured out how to make this whole thing work. As I mentioned, findbugs report generation itself requires that the source code be compiled, so I guess you’d have to invoke Maven like mvn clean compile site install or something. This is just ridiculous.

• In FindBugs, you can specify what rules you want to enforce and what rules you want to ignore. You describe this in a filter file. In a multi-module project, it tends to be more convenient to have just one filter file that all your modules use, rather than having many similar filter definitions. But this seemingly typical use case just doesn’t work with the Maven FindBugs plugin, because the path you specify in the filter file configuration is always interpreted relative to the current Maven module, and there seems to be no way to have it point to the base directory of the project (the ${project.basedir} macro also expands to the current module’s base directory, which is useless.) The documentation does talk about this and gives you a work around. As a Maven plugin developer myself, I understand where they are coming from, but as a Git user, the assumption that requires such a cumbersome workaround (of being able to check out and build modules individually, like you can in Subversion) is unnecessary, yet I still have to pay all the price. This doesn’t make sense.

I’m sorry to say this, but this is a disaster. Integrating FindBugs in Ant project, generating HTML report, and failing a build in case of significant error is fairly straight-forward, and takes maybe 10 or 20 lines at max. But here in Maven, it takes more lines in your POM, not to mention one whole Maven module just for the filter file, plus all these pitfalls. And it still doesn’t attain my original goal of making critical FindBugs issues fail the build.

Experiences like this made me really want to switch to Gradle, but alas, it’s no longer my call alone to make changes like that. So for the time being, I think I’m going back to my good trusted Maven antrun extended plugin. At least it works. And Stephen, this is why Ant fragment is actually more maintainable than the magical combination of Maven hacks.

Uncategorized findbugs, maven