Kohsuke Kawaguchi

There’s an on-going spam problem in Jenkins wiki. We have capture, but either that was broken, or more likely, spamming is done manually.

One of the suggestions was to integrate StopForumSpam service. To that end, I wrote a simple Java client library for their API. The usage would be something like this:

    for (Answer a : new StopForumSpam().build().ip(ip).email(email).query()) {
        if (a.isAppears()) {
            LOGGER.warning("Rejecting, likely spam: "+a);
            throw new UserError("Due to the spam problem, we need additional verification for your sign-up request. Please contact jenkinsci-dev@googlegroups.com");
        }
    }

My hats off to them for providing a great service, and here is hoping that this will help make the dent. We’ll see.

potd stopforumspam

Here’s my 2nd after-JavaOne “project of the day” Jenkins plugin. This has been in the back of my mind for quite some time, but it took this gentleman to grill me on this feature during JavaOne for me to finally put it together — so thank YOU for doing that. although I didn’t catch your name.

The plugin is called the “patch parameter” plugin. This plugin lets you submit a patch when scheduling a build, and that gets applied to the checked out source tree before a build would commence. Any failure to apply a patch will result in a build failure.

This plugin can be used for a “pre-tested commit” workflow. You can work on a change locally, have the diff tested on the server, then if you are satisfied, you can commit it. I can imagine this would be also an interesting building block for integration with code review tools.

I should also note that we have the Subversion Merge plugin for a different approach to a similar problem, and distributed VCS can generally do this better (for example I do one for Git in CloudBees Validated Merge plugin.)

jenkins, potd diff, javaone, patch

Today, I wrapped up the project I started Friday and released iOS device connector plugin.

In addition to listing all iOS devices, this plugin lets you deploy IPA files from anywhere (in Jenkins build via a build step or outside via CLI) to any of the connected iOS devices.

There are still some loose-ends that I’d like to get some feedback on:

• You should be able to select a device without electing a specific UDID, and I’m curious what kind of criteria people needs here. Do you need to be able to say “deploy my app on iPad2 with 16GB memory”, or is “deploy my app to an available iPad” suffice?
• For this to be useful for automated test execution. I think we need to be able to launch the app (and I assume the tests embedded in the app itself will run and shuts itself down in the end?) How do people do it?
• Don’t we need to inject the coordinates of the actual device the app was deployed to, so that your build can do more stuff with it?
• What are other interesting operations to devices. File access? Screenshot?

Anyway, the plugin is already released. I hope you find it useful.

jenkins, potd ios

I was talking to my colleague Mark Prichard about mobile development with Jenkins, and I came up with this idea.

If you are doing iOS app testing with real devices, you need to tether the device with a computer so that you can push the app-to-be-tested to the device. In a local development environment, you’d naturally do this by hooking up a device to your laptop.

But in the CI environment, we can do better. Imagine if you have a Jenkins master connected to a dozen Mac minis, where each Mac mini is then connected to various iOS devices. There’s nothing stopping Jenkins from listing up all the devices connected to all the slaves. Then based on this information, Jenkins can lease devices to builds.

Say someone wants to run a test somewhere. Jenkins can find an unused device and push it, then inject the coordinate of that device to the build. Or how about the axis in a matrix project that allows you to say “I need this test to run with iPhone 3GS,4,5 and iPad 2,3″ then Jenkins will run 5 runs of the tests in parallel?

To that end, as a post-JavaOne fun project, I developed a plugin that lists up all the iOS devices connected to Jenkins.

And just to show that we can list up various properties of this device:

I should be able to combine this with something like fruitstrap so that you can deploy apps from anywhere to any device connected to your Jenkins build farm.

What do you think? Shouldn’t this be interesting? Any thoughts?

jenkins, potd ios, jenkins, plugin

I’m not sure about your Confluence, but my Confluence was dog slow. Page rendering regularly took a second or two, or even worse. That’s why I wrote this plugin.

This Confluence plugin generates static HTML files out of your Wiki pages. It happens every time when someone updates a page, post a comment, add a label, and so on. Anything that can affect the way the page looks in Confluence, and the cache gets regenerated.

Unlike Auto-export plugin, this plugin doesn’t try to apply a different theme, or help you make them look like non-Wiki sites, because the goal is the opposite. I want to make Confluence look faster, as opposed to generating website out of Confluence. To this end, HTML files generated by this plugin is a straight capture of what Confluence is rendering.

These generated files can be then served from frontend Apache reverse proxy. With the use of `mod_rewrite`, Apache will act as a smart caching proxy; it’ll serve static HTML files if they exist, and otherwise it’ll forward the requests to the backend Confluence.

The end result is that your visitors won’t even notice that your Confluence is running with cache. They see the same-old Confluence UI, and the same URLs will keep working. The only difference is that pages magically load a lot faster.

See the live instance to get the feel of how this cache works mostly transparently, and if you want to try this on your own Confluence, the code is here.

potd cache, confluence, wiki

I posted the other day about Groovy’s SecureASTCustomizer and how it is harmful. In the end of that post, I suggested that doing the check dynamically would work better.

So in this installaition of “Project Of The Day”, I went ahead and implemented it. The result is Groovy sandbox.

My main idea is to confine the sandboxed script into its local object graph. The script should be allowed to mutate this graph all it wants, but it’ll be only allowed to include objects of known whitelisted “safe” types (such as String, List, Date etc.), and a few known safe instances, which acts as a bridge between the sandbox and the rest of the world. These bridge objects would have to be written carefully.

To use this, you have to add SandboxTransformer to your CompilerConfiguration first:

def cc = new CompilerConfiguration()
cc.addCompilationCustomizers(new SecureTransformer())
def binding = new Binding();
binding.robot = robot = new Robot();
def sh = new GroovyShell(binding,cc)

And now any script compiled via the resulting shell object will be sandboxed.

When a sandboxed script executes, all of the following operations are intercepted.

• static/instance method invocation foo.bar(....)
• object allocation new Foo(...)
• property access and assignment zot=foo.bar / foo.bar=zot
• attributes access and assignment zot=foo.@bar / foo.@bar=zot
• array access and assignment zot=foo[bar] / foo[bar]=zot

To examine those calls and reject some of them, create your own implementation of GroovyInterceptor and registers it to the thread before you start executing the script:

def sandbox = new RobotSandbox()
sandbox.register()
try {
    sh.evaluate("robot.leftArm.move()")  // this is allowed to complete
    sh.evaluate("robot.selfDestruct()")  // no!
} finally {
    sandbox.unregister()
}

See the robot example for a complete example.

Now let’s see if I can get some feedback from real Groovy experts, and see if they’d be willing to take this into Groovy itself…

potd groovy, sandbox, security

Today’s project of the day is a package renamed ASM library.

I previously wrote about a problem in otherwise quite useful ObjectWeb ASM library. Namely, it breaks backward compatibility in such a way that badly breaks apps/libraries that use them. In that post, I wrote about two proposals to fix the pain point. One is to include debug information, which has been fixed starting 3.x. But the other package renaming hasn’t been addressed in the last 2 years.

This has been in the back of my head, but it never came high enough until recently I had another NoSuchMethodError caused by ASM3. One of the servlet containers shipped ASM3 and it broke Jenkins that bundles ASM2. Between this and ASM4 release for JavaSE 7, which will likely gain popularity over time, I finally decided to fix this problem once and for all, in a way that everyone else can reuse.

The solution, as explained in the original post, is to put each major ASM version in its unique package name. I pakage-renamed ASM2 in org.kohsuke.asm2, ASM3 in org.kohsuke.asm3, and ASM4 in org.kohsuke.asm4. The package name only contains the major version because I trust the ASM developers to maintain compatibility between minor releases (and I believe they’ve maintained this thus far.)

These artifacts are available in org.kohsuke:asm2:2.2.3, org.kohsuke:asm3:3.3.0, and org.kohsuke:asm4:4.0 — these are packaged renamed by jarjar and I tested them somewhat to make sure it’s not downright broken.

If library A depends on asm2 and library B depends on asm3, and someone else uses both A and B, everything will work fine because asm2 and asm3 are in the different pcakages. If A depends on one version of asm3 and B depends on another version of asm3, then the transitive dependency resolution will pick up the newer version and both will work (or you end up implicitly picking up one version over another, and you don’t enjoy the latest bug fixes, but at least it won’t die with LinkageError.)

When you search “asm3″ in Maven central, you see a large number of renamed ASM3 in various projects. Hopefully that madness will stop now!

The other interesting thing about this effort is that I’ve used Gradle to package rename them. Lately I’ve been using Gradle for publishing transformed artifacts like these to a Maven repository, and I like it a lot. But more about that in another post.

potd, Uncategorized asm

Today’s project of the day is a little tool that checks problems in your package names that can bite you on Windows.

For backward compatibility with MS-DOS (which had ended its life more than 15 years ago!), Windows doesn’t allow you to create files/directories of certain names, such as “AUX” or “PRN” (see Microsoft Knowledge Base.)

This can bite those of us who develop on Unix — when you accidentally create a file or a directory of this name (say in the package name), it’ll be all right for you, but your colleagues who use Windows will not be able to build them. Worse yet, if none of your colleagues use Windows either, this problem can get unnoticed until you ship/test the code.

So I wrote a little tool that proactively checks for this problem. It’s an annotation processor that kicks in automatically, so you just need to add it in your classpath when you compile your source files. If you use a build system that can handle transitive dependencies (such as Gradle or Maven), you just need to use it in one of your libraries, and all the code that uses your library also benefits from this check.

My hats off to Microsoft for their rigorous commitment to the backward compatibility, and just one more reason why you should be always testing your Java program on multiple platforms…

potd compatibility, javac, windows

I released Jenkins ChromeDriver plugin that auto-installs chromedriver on all slaves. Here is why this is useful.

ChromeDriver is needed to run WebDriver-based tests with Chrome. Unfortunately this is a native program, so in a heterogenous Jenkins, you need to make sure you have the right binary available on each of the slaves. While that’s not hard, it’s awfully tedious. This plugin fixes that by automating it.

This plugin would be a nice companion to the recently upgraded Selenium Grid plugin that automatically runs Selenium Grid on top of your Jenkins slaves. When you combined, you just install two plugins to the Jenkins master, and without touching slaves at all, you got the whole cluster instantly ready for Selenium2 WebDriver testing. How is that?

jenkins, potd chromedriver, selenium

I was working on Hudson yesterday which led me to develop this little tool called Bridge method injector.

When you are writing a library, there are various restrictions about the kind of changes you can make, in order to maintain binary compatibility.

One such restriction is an inability to restrict the return type. Say in v1 of your library you had the following code:

public Foo getFoo() {
    return new Foo();
}

In v2, say if you introduce a subtype of called FooSubType, and you want to change the getFoo method to return FooSubType.

public FooSubType getFoo() {
    return new FooSubType();
}

But if you do this, you break the binary compatibility. The clients need to be recompiled to be able to work with the new signature. This is where this bridge method injector can help. By adding an annotation like the following:

@WithBridgeMethods(Foo.class)
public FooSubType getFoo() {
    return new FooSubType();
}

… and running the bytecode post processor, your class file will get the additional “bridge methods.” In pseudo-code, it’ll look like this:

// your original definition
@WithBridgeMethods(Foo.class)
public FooSubType getFoo() {
    return new FooSubType();
}

// added bridge method
public Foo getFoo() {
    invokevirtual this.getFoo()LFooSubType;
    areturn
}

Such code isn’t allowed in Java source files, but class files allow that. With this addition, existing clients will continue to function.

In this way, you can evolve your classes more easily without breaking backward compatibility.

For more about how to use it in your Maven project, the project website.

potd