Internet Explorer only takes string in HTML5 postMessage

September 11th, 2012

Today I learned the hard way that our beloved Internet Explorer doesn’t handle HTML5 cross-domain messaging functionality (AKA postMessage) correctly.

I have done a reasonable amount of research before I invested a significant time into this, and so I knew it doesn’t postMessage across tabs and windows beforehand. This limitation was not a problem for me, because I was going to use this in IFRAME.

The problem that I discovered much too late is that IE only accepts a string as a message (see BSTR in a signature? That’s a Windows speak for passing UTF-16 strings.) This is despite the fact that the draft spec allowed any object and not just strings as early as Novemember 2010 .

And other sites like caniuse.com and various blog posts failed me too, as they failed to mention this limitation.

So I’m posting this here in the hope that this will prevent other people from wasting time.

AAARRRGGGGGGGHHHHH!!!!

Uncategorized , , ,

Jenkins Git Server Plugin

August 31st, 2012

Jenkins Git Server plugin is a so-called “library plugin”, which doesn’t offer any user-visible feature by itself, but instead enables other plugins to do something easily inside Jenkins. In case of Git server plugin, it allows other plugins to easily embed Git server functionality (via JGit) — create/manipulate Git repositories in the Jenkins server, expose it via SSH and HTTP transport for push/pull, and maintain local check out of those repositories.

As an example of how to use this plugin, I wrote Git userContent plugin. This plugin exposes the $JENKINS_HOME/userContent directory as a Git repository, and enables administrators to use git to push/pull changes and manage them with history.

In terms of code, there are two classes that plugins like git-userContent-plugin should be interested in.

One is HttpGitRepository, which represents Git repository access via HTTP. Typically you have some directory inside $JENKINS_HOME that houses the repository, then you subtype GitHttpRepository and override abstract methods to fill in the missing details. FileBackedHttpGitRepository is a convenient default implementation that simplifies this further. GitUserContentRepository in git-userContent-plugin is an example of using this class. This use also implements RootAction to bind this repository at http://server/jenkins/userContent.git, and I expect this combination to be fairly common.

The other class of interest is RepositoryResolver. Git server plugin adds necessary Jenkins SSH CLI hook for exposing Git repositories over SSH. The only missing link here is that when the client runs “git clone ssh://server/foo/bar/zot.git“, we need to figure out what repositories on the server corresponds to /foo/bar/zot.git, and that’s what the RepositoryResolver extension point does. The sample implementation in git-userContent-plugin will be hopefully self-explanatory. In this case, GitUserContentRepository is a singleton (because it’s RootAction), so we inject that and basically just delegate the calls to it.

I’m looking forward to seeing more plugins take advantages of this feature to expose data over Git repository. I think there’s a lot of interesting uses to it.

jenkins

Meetup “Automated testing & continuous deployment for mobile apps” tomorrow

August 29th, 2012

Mark Prichard from CloudBees, myself, and Matt Solnit from SOASTA will deliver a joint talk Thursday evening at Silicon Valley Cloud Computing Group.

Our part of the talk will take an iOS application and a backend Java server, then discuss how you can set up the pipeline of continuous build and test for them on Jenkins. The demo will be based on the CloudBees platform and we think it’s easier to do it that way than setting everything up on your own, but the setup should equally apply to your local Jenkins installation.

The part of the talk from Matt highlights SOASTA’s CloudTest product, which is basically a robot tha emulates user interaction on the device. My understanding of this is that it’s similar to Frank and KIF, but it comes with a recorder that lets you do “programming by example” — you manually interact with the test target app, and the recorder will turn that into a test scenario (Selenium IDE, anyone?)

Apple has locked down the iOS application development and distribution which makes the automated testing setup more painful than it should be, but because of that, I think a talk like this that shows the whole setup is worthwhile.

The event will be Thursday evening, at Yahoo! Campus in Sunnyvale. Please join us if you can. The event is free but requires RSVP. Beer and pizza are provided.

jenkins , ,

Jenkins User Conference Israel

July 1st, 2012

I’ll be visiting Israel this week, for Jenkins User Conference in Herzliya. I think this is my 3rd time visiting there, and I always enjoyed my visit and people over there (it really is an interesting place in many ways that everyone should visit once!) If you live in Israel, you can still register for the event!

You can see the planned talks, but there’ll be a lot of good “show and tell” talks where seasoned users share their experiences and setups to others, as well as more “emerging technologies” kind of talks, where people can get the idea of what’s cooking.

I’m planning to do a talk that contains a little bit of both; a bit about features that I developed for Jenkins that I use myself (some with my CloudBees hat on), then more about what the community has been working on in the core lately, and what I’d like to work on in near future.

It never ceases to amaze me that this piece of software called Jenkins has grown so much that we have 100s of people using it everywhere we go. Looking forward to meeting you all, and don’t forget to demand Jenkins stickers from me!

jenkins ,

POTD: Confluence static cache generator plugin

June 22nd, 2012

I’m not sure about your Confluence, but my Confluence was dog slow. Page rendering regularly took a second or two, or even worse. That’s why I wrote this plugin.

This Confluence plugin generates static HTML files out of your Wiki pages. It happens every time when someone updates a page, post a comment, add a label, and so on. Anything that can affect the way the page looks in Confluence, and the cache gets regenerated.

Unlike Auto-export plugin, this plugin doesn’t try to apply a different theme, or help you make them look like non-Wiki sites, because the goal is the opposite. I want to make Confluence look faster, as opposed to generating website out of Confluence. To this end, HTML files generated by this plugin is a straight capture of what Confluence is rendering.

These generated files can be then served from frontend Apache reverse proxy. With the use of `mod_rewrite`, Apache will act as a smart caching proxy; it’ll serve static HTML files if they exist, and otherwise it’ll forward the requests to the backend Confluence.

The end result is that your visitors won’t even notice that your Confluence is running with cache. They see the same-old Confluence UI, and the same URLs will keep working. The only difference is that pages magically load a lot faster.

See the live instance to get the feel of how this cache works mostly transparently, and if you want to try this on your own Confluence, the code is here.

potd , ,

Jenkins User Conference New York: my take

June 13th, 2012

In one of those days, I’ll get a small enough computer whose battery won’t die in 45 minutes, but until then, my apologies for the belated Jenkins User Conference travel report.

So, as we tour around the world bringing Jenkins User Conference near you, the last stop was New York City.

I’ve kicked off the day by recaping the community activities in the past year, including Subversion 1.7 support, new UI effort, and several new extension points in the Matrix project. From my day job side, I’ve introduced BuildHive, a new addition to CloudBees DEV@cloud that builds your GitHub repositories in a few clicks, and a new version of Jenkins Enterprise by CloudBees with high-availability features.

There are two parallel tracks that went through the whole day, including talks from people who has been around in the community for a long time. Just to name a few, there’s Monty Taylor from HP discussing how OpenStack deploys Jenkins and Gerrit to ensure that the trunk never breaks, and Mike Rooney giving advices on running mission-critical Jenkins instances. Jesse Farinacci was also there, even though we couldn’t convince him to give a talk (and you can see his recap post in jenkins-ci.org.)

But aside from those who I already knew, there were many talks from whom I call “super Jenkins admins” — those who not only deploys Jenkins for their orgs, but push it to 11, by writing custom plugins, changing the visualization, writing peripheral programs that interact with Jenkins, etc.

For JUC NY, the super Jenkins admin award has to go to Jesse and David from AtTask. There are a number of awesome things they’ve done. First, they completely took the Jenkins instance to cloud, and not only do they provision build slaves from EC2, but also provision the entire Selenium Grid and its remote agents as well. Then they implemented a custom view plugin so that developers can see which stage in the pipeline their changes are at. They’ve also integrated Jenkins to their own product, AtTask, and automatically turns test failures into tickets (and it’s smart enough to reopen the old ones when a regression occurs!)

One thing I felt as I was listening to these incredible user story talks is that it’d be nice if we have means to capture the configuration of those instances and share those with others. Kind of like how you share cooking recipes. I’m not exactly sure how to achieve it, but it should include a list of plugins, some configurations, and it’d let you import those into your instance, so that you can replicate it, play with it, and modify it easily.

And last but not least, The talk from Noah about Jenkins REST API was also very good, in that it really highlighted a part of Jenkins that’s not widely used, and it gives everyone something that they can take back to their home.

I hope this post encourages you to come to future JUCs. Please submit papers, and register from the website. I’m really looking forward to the next one in Israel, and the one after that, Tokyo, has already signed up 700 people, so it’ll be a crazy event!

Uncategorized

Push changes directly into BuildHive, and never run tests again!

June 5th, 2012

On top of pull requests auto-build, BuildHive now allows you to push changes directly in via ssh. I call this feature “validated merge”.

If you are an active developer of a repository, chances are that you don’t use pull requests to send in changes. You probably just push changes directly into your repository instead. But one of the common problems of doing this is that if your change breaks the build, you are going to discover it too late. With validated merge, this problem will never happen again, and this is how it works.

First, you add the buildhive user as a collaborator to your repository, so that it can push when the build is done. I didn’t feel comfortable doing this automatically, so you need to do it explicitly.

Second, login to BuildHive (just once is suffice, so that it learns your public keys you registered with GitHub), head to the job page in BuildHive (like this), and click “Git repository for validated merge” link from the left. Click the copy button to copy the “git remote add” command, paste that in your shell, and execute that on your local repository. This will add BuildHive as a remote repository.

Preparation is now complete. Go modify code like you normally do by creating commits, and when you are ready to push, push your changes to “jenkins” instead of pushing to “origin”. If you are on a branch, replace “master” with the branch name you are working on:

$ git push jenkins master

BuildHive will check out the current tip of the specified branch in GitHub, merge that with the changes you just submitted, then do the build and run tests. If the merge result successfully completes the build and tests, the result will be pushed to GitHub by BuildHive.

If you are unlucky and your changes weren’t as good as you thought, you’ll learn that it didn’t pass the builds. You then stash your work, come back to the broken changes, and rework those. You probably don’t remember which commit you pushed to BuildHive, so BuildHive provides a tag you can fetch into your workspace to get to it.

$ git stash
$ git fetch -n ssh://anonymous@buildhive.cloudbees.com/kohsuke2/sandbox-ant tag changes/45
$ git checkout changes/45
$ … make edits ...

You can add more commits to correct the problem, or you can even amend your commits — since your changes didn’t land in the repository, amending, rebasing, and so on is just fine. You can then re-push your changes and then go back to what you were working on before interrupted by a failure.

$ git push jenkins HEAD:master
$ git checkout master
$ git stash pop
… resume the work you were doing ...

If you’ve already headed home, your colleague can do this workflow for you.

Your time is precious. Don’t waste it by watching your laptop to complete tests. Instead, let BuildHive do it for you!

By the way, this is a feature already available in Jenkins Enterprise by CloudBees, so if you like the workflow but your source code isn’t in a public GitHub repository, you can do the same thing with any Git repository inside your firewall.

jenkins , ,

Jenkins Pry Plugin

May 31st, 2012

I released Jenkins Pry plugin 1.1 today. This plugin adds the “pry” command to Jenkins CLI so that you can use pry to introspect a running Jenkins instance via pry.

Because this builds on top of Jenkins CLI, you can remotely connect from your laptop to your Jenkins master, safe with SSH public key authentication and transport encryption.

And because this builds on top of pry, it lets you interactively explore a rich object graph of Jenkins, doing bulk operations, checking the state of the system, etc.

I’ve done enough integration work so that the readline support in pry works as expected. So you can enjoy the code completion and command history support in pry, even though it’s running remotely. Nifty, isn’t it?

I hope this is useful for Jenkins admins who are more familiar with Ruby than Groovy (for which we had this “groovysh” CLI command for the longest time), but my real goal is to use this kind of remote Ruby code execution mechanism to let people run tests quickly when they develop Jenkins plugins in Ruby. This morning I’ve used this to verify that I can embed Cucumber tests. So stay tuned for more on that front!

jenkins

POTD: Groovy Sandbox

April 29th, 2012

I posted the other day about Groovy’s SecureASTCustomizer and how it is harmful. In the end of that post, I suggested that doing the check dynamically would work better.

So in this installaition of “Project Of The Day”, I went ahead and implemented it. The result is Groovy sandbox.

My main idea is to confine the sandboxed script into its local object graph. The script should be allowed to mutate this graph all it wants, but it’ll be only allowed to include objects of known whitelisted “safe” types (such as String, List, Date etc.), and a few known safe instances, which acts as a bridge between the sandbox and the rest of the world. These bridge objects would have to be written carefully.

To use this, you have to add SandboxTransformer to your CompilerConfiguration first:

def cc = new CompilerConfiguration()
cc.addCompilationCustomizers(new SecureTransformer())
def binding = new Binding();
binding.robot = robot = new Robot();
def sh = new GroovyShell(binding,cc)

And now any script compiled via the resulting shell object will be sandboxed.

When a sandboxed script executes, all of the following operations are intercepted.

  • static/instance method invocation foo.bar(....)
  • object allocation new Foo(...)
  • property access and assignment zot=foo.bar / foo.bar=zot
  • attributes access and assignment zot=foo.@bar / foo.@bar=zot
  • array access and assignment zot=foo[bar] / foo[bar]=zot

To examine those calls and reject some of them, create your own implementation of GroovyInterceptor and registers it to the thread before you start executing the script:

def sandbox = new RobotSandbox()
sandbox.register()
try {
    sh.evaluate("robot.leftArm.move()")  // this is allowed to complete
    sh.evaluate("robot.selfDestruct()")  // no!
} finally {
    sandbox.unregister()
}

See the robot example for a complete example.

Now let’s see if I can get some feedback from real Groovy experts, and see if they’d be willing to take this into Groovy itself…

potd , ,

Groovy SecureASTCustomizer is harmful

April 27th, 2012

I was looking at Groovy DSL slides from Guillaume Laforge when I noticed about SecureASTCustomizer, which led me to what appers to be the original introduction post from Cedric.

Being able to lock Groovy execution down would enable me to use Groovy in more places, so I did a bit of experiment. But I regrettably have to conclude that this feature is practically unusable. In fact I’d argue that it is actively harmful, as it gives a programmer a false comfort.

The fundamental problem is that Groovy is a dynamic language, yet SecureASTCustomizer works by looking at Groovy AST statically. So it’s very easy for Maloney, a malicious attacker, to bypass many of the checks. For example, Cedric’s post talks about how it can let you blacklist/whitelist classes that can be imported. Well, the actual goal of the programmer is to prevent the class from getting used, and not to get them imported. And sure enough, even if I white list the importable classes to java.lang.Math, Maloney can still do Math.class.forName('some.secret.class') to get a reference to a Class, and therefore render the import restrictions pointless.

Then I thought about disabling access to the getClass() method. But this doesn’t work well either because Groovy allows 5."class" and 5["class"] to access properties. To statically prevent this, you’d have to prohibit the array access and a string literal, but that doesn’t leave much of a language!

Many other checks offered by SecureASTCustomizer are equally useless. For example, there’s receiversClassesWhiteList that’s supposed to let you restrict the methods the script can invoke by whitelisting the declaring class of the method. But once again, this is a static check! Groovy compiler doesn’t work very hard to infer types, so much so that it can’t even guess that x=="foo" is a boolean type. Therefore, if you actually try using receiver whitelisting, pretty quickly you’ll discover that you either have to allow Object as a receiver (because Groovy assigns this to every expression when it couldn’t infer the type), which will basically renders the point of whitelisting moot as you can now invoke any method by simply casting the expression to Object.

If you go the other route and disallow Object as a receiver. That will reject almost all non-trivial scripts. Or I suppose you can prohibit a method call, but that doesn’t leave much of a language, does it.

Like I said, I think this is fundamentally a futile approach. You just can’t perform any meaningful static sandboxing on a dynamic language.

Instead, what I think is more fruitful is a dynamic checking. For example, what if the compile-time AST transformation intercepts every method call and property access? That is, transform z=x.y as z=checkedGet(x,"y"), transform x.y=5 into checkedSet(x,"y",5), and finally transform o.foo(a,b,c) into checkedCall(o,"foo",[a,b,c]). This does make execution a whole lot slower, but I can now perform meaningful checks. And unlike Java SecurityManager, this is a lot more friendly to libraries and web applications, who cannot take over the entire JVM.

I haven’t actually put together such an AST transformer, but this doesn’t look too hard.

What do people think?

Uncategorized , ,