Introducing template plugin

The Nectar 11.10 release that we just made has the template plugin, which I think is one of the very useful plugins that we added to Nectar that’s not available in open-source Jenkins. So I wanted to talk about it a bit.

Many serious Jenkins users have had this common problem that they’ve got a lot of jobs that look similar. JENKINS-3157, which is currently the most voted issue in JIRA, speaks to that same pain point. The template plugin is my answer to this problem.

What this plugin does, is to allow you to define a “model”. A model is an abstract concept that fits your particular problem domain. You do this through defining a set of attributes, kind of like how you define a class with a set of properties in Java, or how you define a table with a set of columns in database. You then define a transformation, which converts this model into something that Jenkins understands.

Creating a template

This is all rather abstract, so let’s look at a concrete example. We are going to define a hello world builder template, which lets you enforce the standard way to greet someone. You can then use this template to greet a lot of people in many jobs.

Installing this plugin, you get the “templates” menu on the left. You can click that and create a template:

You create a builder template from the same faimilar wizard you use to create a job:

Then here is the main meat.

There are two main things to configure here: attributes and a transformer.

When you define a template, you first ask yourself “what do I want my users to enter when they use my template?” The answer to that question becomes attributes. In this hello world builder, we want the user to configure who we are saying hello to, so we define one attribute named “target” for this purpose. The user should see the single text field for this, so we choose the type accordingly. The display name and inline help are self-explanatory. They control what the user will see when they are editing their free-style projects to use our new builder.

The second question you should ask when you define a template is “how does it execute?” (or more generally “how does it map to the terms Jenkins understands?”) The answer to that question becomes the transformer. In this tutorial, our builder will turn into a shell script that says hello (whereas your real template would probably turns into a shell script that gets some real work done, like building a component, running a test, etc.) So we’ll choose “generate a shell script to execute via Groovy”.

In the text area, we’ll fill the shell script that this build step is going to execute, but with expressions here and there of the form ${...} (because this is a Groovy template). ${target} refers to the actual value of the target attribute we defined above, and ${build.fullDisplayName} is a Groovy expression to access the getFullDisplayName() method (which returns the full name of the build) of the build object, which refers to the current build in progress. The ${build.fullDisplayName} needs to be quoted because this is going to look like test #1, and # gets interpreted by shell as a comment sign unless you quote it.

Using a template

Now that we have our first builder template, let’s create a free-style project that actually uses it. Go back to the Jenkins top page, and create a new free-style project. You’ll be taken to the configuration page. This part of Jenkins should be already familiar to you.

When you click “add build step”, you should see the newly created “Say Hello World” builder. You click it, and you see the say hello world builder added to your project. And you see the “target” attribute you defined in the configuration page.

I configured this to say hello to Kohsuke. If we save it and run it, lo and behold, you’ll see that it works.

[workspace] $ /bin/sh -xe /tmp/
+ echo Hello to Kohsuke from test #1
Hello to Kohsuke from test #1
Finished: SUCCESS

This allows you as the build guy to create and enfroce a certain way your stuff gets built/tested, and your users get shielded from all the gory details of how the build/test actually get done. You can create a layer between Jenkins and your users, and make Jenkins talk in the language your users understand, instead of making them talk in the language Jenkins understands. This is one of the points of the template plugin.

Changing a template

Now, let’s change the definition of the template, and see how it affects the instances.

We’ll go back to the template definition by going back to the top page, clicking the “Templates” link in the left, and clicking the configuration icon on the right to the “Say Hello World” template.

Instead of saying hello, now we make it say good evening.Click “save” to save this new definition:

Now, when you update the template definition, all the use of this template automatically reflects the change you made. So without revisiting the configuration page of the freestyle job, let’s simply schedule another build and see its console output:

[workspace] $ /bin/sh -xe /tmp/
+ echo Good evening to Kohsuke from test #2
Good evening to Kohsuke from test #2
Finished: SUCCESS

This is another point of the template plugin — when you change the definition, all the uses of this template get updated right away. So if you need to tweak how those high-level abstract concepts map to the lower level command sequences, you only need to edit it once and save it.

Try it out for yourself

If you like what you’ve read so far (I hope you do), please play with this. You can either download the whole Nectar and play it in a separate sandbox, or you can enable the secondary update center to install these plugins on your OSS Jenkins deployments.

And there’s a lot more to the template plugin, which I hope to write about in a few days.

Quiz answer: memory leak in Java

I posted a little quiz yesterday, and here is the answer.

The short answer is that InputStream needs to be closed. It’s easy to see why if it’s FileInputStream because you know the file handle needs to be released. But in this case, it’s just ByteArrayInputStream. We can just let GC recycle all the memory, right?

Turns out GZIPInputStream (or more precisely Deflater that it uses internally) uses native zlib code to perform decompression, so it’s actually occupying more memory (about 32K-64K depending on the compression level, I believe) on the native side, while its Java heap footprint is small. So if you allocate enough of those, you can end up eating a lot of native memory, while Java heap is still mostly idle. Even though those GZipInputStreams are no longer referenced, it just doesn’t create enough heap pressure to cause the GC to run.

And eventually you eat up all the native memory, and zlib’s malloc fails, and you get OutOfMemoryError (or your system starts to swap like crazy and your system effectively becomes unusable first.)

The other interesting thing to note is that -XX:HeapDumpOnOutOfMemoryError doesn’t do anything in this case. I read the JVM source code and I learned that heap dump only happens when OOME is caused during 3 or 4 specific memory allocation operations, like allocating a Java object, array, GC saturation, and a few other things. There are many other code passes in JVM that reports OOME, like this zlib malloc failure, that doesn’t trigger heap dump. There’s no question HeapDumpOnOutOfMemoryError is useful, but just beware that in some cases it doesn’t get created.

I knew that GZipInputStream is using native code internally, but I didn’t think about it too much when I was putting this original code together. Humans can’t think about all the transitive object graph and its implications.

The other lesson is that now I know why ps sometimes report such a big memory footprint for JVM while jmap reports only a modest usage. The difference is native memory outside Java heap, although unfortunately I don’t think there’s any easy way to check what’s eating the native memory.

My colleague and friend Paul Sandoz pointed out that if GZipInputStream was nice enough to free them up at EOF, it would have saved a lot of hassle, and I think he’s right — one still needs to consider the case where IOException causes the processing to abort before hitting EOF, but it would have helped, because those abnormal cases would be rare. I mean, there’s no harm in doing so, and anything that makes the library more robust in the face of abuse is a good thing, especially when the failure mode is this cryptic.

Quiz time: memory leak in Java

Today I had an interesting debugging exercise, and I felt like I learned a new lesson that’s worth sharing with the rest of the world.

I had the following code, which takes a small-ish byte array and deserializes it into an object (let’s say someNotTooBigData is something like new byte[]{1,5,4, ... some data... }.) Seems innocent enough, no?

voidObject foo() {
	byte[] buf = someNotTooBigData();
	return new ObjectInputStream(new GZIPInputStream(
	    new ByteArrayInputStream(buf))).readObject();

But when this is executed frequently enough, like while(true) { foo(); }, it creates OutOfMemoryError. Can you tell why? I’ll post the answer tomorrow.

Jenkins Community Survey

There’s currently a survey running to get a better sense of our use base. Those inputs help us steer the effort wisely, so we appreciate your taking time to fill it in. The result would be more useful if larger number of people participate, so feel free to encourage others to fill it in as well.

(The interesting thing about an open-source project is that you don’t get to tell other people what to work on — you can only try to influence them, and data like this is very useful.)

In the same spirit of the fundraising drive in NPR, CloudBees is throwing in a “thank you gift” of AppleTV as an added incentive. See their blog post for more details.