Support API freedom

I was reading this article from Steve and Sacha about the API copyrightability, and found myself in a violent agreement. If you haven’t read it, I highly recommend it.

For those of you who haven’t been following the tech news, the issue at hand is Android — Google neatly side-stepped Java’s compatibility requirements by introducing a new runtime/VM and said that Android is not Java. Oracle sued Google claiming that the Java API is a copyrightable material, and that Google can’t just create a whole new implementation that’s API-compatible with Java. Oracle lost the case, but now Oracle is appealing, and they are garthering legacy vendor friends to argue that API not copyrightable is bad for economy.

But wait, surely more competition is bad for them vendors, but what about the instant gain we the developers got when Android came along, in becoming instantly productive in this entirely new platform?

Looking at the comment secion of the article, I was bit disappointed that some people saw this only as a storm in a teacup, or that this is an issue only about Java, and their favorite programming ecosystem (C#, Ruby, …) are OK. But it’s quite a contrary. If the appeal is successful, it has a broad implication on all sorts of APIs. As they say, first they come for the communists, and you think you are safe, but by the time they come to you, it might be too late!

Take Mono for example. Sure, C# and CLI are under Microsoft Community Promise. But what about the vast APIs in the .NET Framework, which is necessary for writing any meaningul application? What about all the Win32 APIs that Wine implements? Or how about Eucalyptus implementing the Amazon Web Services API? Sure, they might be in a good relationship now, but what if IBM acquires Eucalyptus and started a cloud offering of the same API?

As a developer I benefit every day from the compatibility and being able to migrate from one vendor to another without losing everything. And when I look back at PC/AT, x86 instruction sets, Java EE APIs, and so on, I truly believe that the openness is good not just for us the developers but for the broader economy as a whole.

So after reading the article, I felt like I wanted to help the cause and voice my support, but I wasn’t sure how — I’m just a developer and not a lawyer. So I created a White House petition. Not so much because I expect the White House to do something about it, but it’s a good enough neutral petition site that hopefully people feel safe enough to join. If you agree with the cause, please join the petition and help spread the words, so that our voices get heard.

Writing programs that drive Jenkins

One of the interesting discussions during SCALE 10x was about using Jenkins as a piece of a bigger software. This person was interested in using Jenkins to run some business analysis operations, and wanted to have a separate interface for business oriented people. This is actually an emerging but common theme I hear from many users. Another company in San Jose actually built the workflow engine that uses Jenkins as a piece in a bigger application (aside from the actual build and test, this workflow involves reviews, approvals, etc.), and GitHub Janky can be classified as one such app, too.

This is something I always believed in — that every piece of software needs to be usable by a layer above. Or put another way, every software should be usable as a library.

So in this post I’m going to discuss various ways you can programmatically drive Jenkins.

Let’s start with the REST API of Jenkins. For most of the data Jenkins renders as HTML, you can access its XML version and JSON version (as well as a few other formats, like Python literal fragment.) You do this by adding /api to the page (see for example.) Those pages discusss other REST API where applicable. For example, you can POST to certain URL and it’ll create/update job definitions, etc.

If you are going to use REST API, you might find the auto-discovery for Jenkins useful. You can discover Jenkins on the local subnet via UDP broadcast, or DNS multi-cast. There’s also a distinctive HTTP header “Jenkins-Version” on the top page of Jenkins that allows your application to verify that it’s talking to a real Jenkins server, as well as an instance ID that allows you to identify Jenkins instanecs. These features allow you to build smarter applications.

For Jenkins protected by some authentication mechanism, you can use the user name + API key in the HTTP basic auth (and I want to add OAuth support here.)

REST API is great that it’s programming language agnostic. It is also convenient that neither the server nor the client has to trust each other. But those APIs are bound by the request/response oriented nature of the HTTP protocol.

Another great integration point for Jenkins is the CLI. This uses the same underlying technology that drives master/slave architecture, which enables your command line clients to be a lot more intelligent. For example, REST API exposes an URL that you can post to get a build started. But the equivalent command in CLI can have you block until the build is complete (and exit code indicates the status), or run the polling first and proceed to build only when the polling detects a change, or allow you perform a parameterized build with multiple file uploads very easily. For protected Jenkins, CLI supports SSH public key authentication to securely authenticate the client.

A slightly different version of the CLI is “Jenkins as SSH server”. Jenkins speaks the server side of the SSH protocol, and allow regular SSH clients to execute a subset of CLI commands. In this way, you don’t need any Java runtime installed on the client side to drive Jenkins.

These two integration APIs are often much easier to script than REST API.

Those APIs are available for non-privileged users, and they are great for small scale integrations. But for more sophisticated integration needs, we have additional APIs.

One is the REST API access to the Groovy console, which allows administrator users to run arbitrary Groovy script inside the Jenkins master JVM (and you can submit this script as POST payload, and get the response back as the HTTP response.) This allows you to tap into all the Jenkins object models. Unlike the REST API, in this way you can ship the computation, so in one round-trip you can do a lot. You can do the same with CLI, which also lets you access stdin/stdout of the CLI.

The other sophisticated integration API I wanted to talk about is the remoting API that does Java RPC (not to be confused with the remote API, which is the synonym for the REST API.) The remoting API is the underlying protocol that we use for master/slave communications, and it revolves around the notion of shipping a closure (and code associated with it) from one JVM to another, execute it, and get the result back. If your application runs elsewhere, you can establish the remoting API channel with Jenkins master, then prepare a Callable object. You can then have Jenkins master execute this closure, and the result is sent back to your JVM.

There’s an example of this available. You bootstrap this in the same way the CLI client talks to the master, then you “upgrade” the communication channel by activating the remote code download support (which requires the administrator privilege, for obvious reasons.)

The great thing about this is that your data structure is rich Java object model all the way, and you never have to translate your data to externalizable serialization data format like XML or JSON. This greatly simplifies your program.

I think this list covers all the major integration APIs that Jenkins offers. If you are building any interesting applications that uses Jenkins as a building block, please share your experience so that we can make it better!

Jenkins now acts as an SSH daemon

Starting in the upcoming Jenkins 1.446, Jenkins speaks the server-side of the SSH protocol. I think this is exciting, and here is why.

For quite some time now, there is Jenkins CLI, which lets you access Jenkins from command line by using a custom client jar file. There are several dozen commands available (that you can check by visiting http://server/jenkins/cli), ranging from creating a job to shutting down Jenkins. These CLIs are often a lot more scriptable than HTTP interfaces, and so it is often used by other programs to interact with Jenkins.

Jenkins 1.446 exposes a large subset of these commands through SSH, making it even easier to access this functionality. You no longer need any special client jar nor Java VM to talk to Jenkins. All you need is a stock ssh client. As I explained earlier, Jenkins lets you register your SSH public keys, and that is how you authenticate.

As always, the server side of this is extensible. Plugins can define additional commands to be exposed via SSH, which is a great building block for more sophisticated integrations. This is particularly so because SSH is a standard transport protocol for many tools, such as Git, Subversion, and so on. I’m planning to write a few plugins that take advantage of this in the coming days.

Wiki article talks more about the details, including how you can discover the endpoint, how you can verify the server key, and so on.

(And no, this is not about making Jenkins a general-purpose SSH server that you can use to login to the server.)

Jenkins Plugin Tip: Access control and visibility in actions

As I discussed before, Action is one of the primary ways plugins use to add more information to the top page, project pages, build pages, and so on.

Today I’d like to talk about writing protected actions that require some kind of access control. Depending on your requirements, there are several ways to do this. There are three parts in Action that affects behaviours with this regard.

One is the getIconFileName() method. As its javadoc states, returning null from this method hides your action from the HTML page. So if you have an action and you don’t want to show it for users who don’t have the permission to access it, it could be something like the following:

public String getIconFileName() {
    return Jenkins.getInstance().hasPermission(Model.LIST)
         ? "gear.png" : null;

The next is the getUrlName() method. This method also allows null as the return value. This might sound similar to getIconFileName() but the implication is different. With getIconFileName() returning null, the rendered HTML page won’t show the link but if someone knows the URL, they can still access it. But with this method returning null, it’s as if no such URL is recognized.

So with the code like the following, if someone without the permission requests it, he’ll get 404 not found (modulo a bug, which I just fixed toward 1.443). This is sometimes desirable, as not only can you hide the data, but you can also hide the fact that something exists. For jobs, Jenkins does this — http://jenkins/job/top-secret-project/ returns 404 not found, not 401 forbidden, so guessing the project name will not help attackers gain any information.

public String getUrlName() {
    return Jenkins.getInstance().hasPermission(Model.LIST)
         ? "template" : null;

But this is sometimes undesirable, as users will not be prompted for authentication. If someone hits the link with an expired session, he’ll get 404 not found and he needs to be smart enough to know that this is because he hasn’t logged in, then navigate manually to the login page and come back. To avoid this problem, you’ll do the following and advertise the URL all the time.

public String getUrlName() {
    return "template";

And that brings me to the third part, because getUrlName() now always returning non-null means you aren’t actually checking if those who are accessing has a permission to do so. To do this, you use StaplerProxy.

class MyAction implements Action, StaplerProxy {
    public Object getTarget() {
        return this;

What happens is that when the URL that someone requests hits this action (or something underneath), we’ll verify that the requestor has the permission. If not, this will initiate the authentication, such as redirecting the user to the login page, or initiating the OpenID protocol with the preconfigured identity provider. Normally this interface is used to defer the UI processing to another object (sort of like how symlink works), but in this case we return this to indicate that we process this request by ourselves, and it’s smart enough not to cause infinitely recursion.

I guess the rule of thumb is that (1) you have to check the permission either in getUrlName() or getTarget(), or else there’s no access control, (2) you use getIconFileName() control the visibility in the HTML page, and (3) you use getUrlName() to control if you want 401 or 404 in case the access is denied.