Trail of Coincidence

A guest was coming to our office over lunch, so food was ordered for everyone. Except it turned out that due to a miscommunication, mine wasn’t there. No problem, I said. Happy accident. I’ll go out for lunch later. Maybe somebody up there is trying to tell me that I should eat out today.

I was just going to a nearby restaurant, but a random lunch conversation led to somebody mentioning Tofu House, my favorite Korean restaurant. Great, I thought. Somebody up there must be telling me that that’s where I should go.

On my way back from Tofu House, there was a road work going on at an intersection where I was normally going to make a right turn. I decided that that’s a sign that I should keep going straight. I can just get back to the office in a different way.

It was a hot day, and this road was going to go right in front of my favorite smoothie shop. Surely that must also be a sign? So I made a stop and got myself a smoothie.

At this point, I was starting to get excited. It started to feel like a series of serendipitous events was happening to me. Is this going somewhere? Wherever that is, I’m feeling lucky today.

I resumed my drive back to the office, and because I was driving a road I don’t normally take, in this one complicated intersection I ended up in a wrong lane that forced me to a wrong road. A series of no U turn and no left turn prevented me from taking an obvious way back. The detour is getting big.

See, something IS happening! This cannot be a coincidence! Maybe I’ll encounter a future soul mate? Love of my life (OK, maybe not, I’m already married.) Save somebody’s life? Find a million dollar?

I looked around. Nothing. I kept driving and looking around. Still nothing. I eventually came back to the office parking lot. The trail of serendipity is gone, abruptly.

I still can’t shake that off from my head as I write this. I was convinced that this was going to take me somewhere. What clue did I miss?

Suddenly, I’m not feeling lucky anymore.

Camping in Livermore

In the Saturday Japanese school my daughter goes to, there are 4-5 families that mine has been particularly close to. With some families relocating and what not, faces have slowly changed over time, but some of us go all the way back to when my daughter was a baby.

Over the memorial day weekend, the group went to an overnight camping together. It’s one of those camps where we all show up with loads of gears, food, and drinks in our cars. Adults would indulge in non-stop eating and conversation around a camp fire, while kids disappear into the nature and enjoy themselves.

The next morning I woke up and my daughter wants me to see a project she and her younger friends built together in a creek nearby. Cool, I guess that’s what they were working on when I didn’t see them. I’ve followed them upstream to a make-shift dam made by rocks. It’s only partially completed — perhaps a few meters from a bank. Satisfied with my wow-ing and ohh-ing, they resumed their project, picking stones and dropping them into the creek, taking turns, self-organized. I aimed my camera in the hope of taking some pictures of their joyful expressions.

That’s when I understood. It just came to me like that.

See, my daughter is right in the middle of teenage years, where they struggle and worry to figure out what’s their passion. My frame of reference, namely how I grew up, is completely useless here because I just run into computers in the 7th grade, I got hooked, and the rest is history. Like countless other parents, naturally this has been at the top of our mind; Exposing her to various extra curricular activities, from astronomy to performing arts; Cutting back to make time & space that she can use to explore and find her passion freely. Somewhere in the back of my mind I knew I was worried too much, but how can you not, when the stake is so high? I didn’t want her to just assemble a LEGO set I bought her. I wanted her to assemble something on her own out of the LEGO bin. That’s so important to me that’s the meaning of her name I gave her.

Well, guess what, while I was having fun with beer in my hand, fiddling with a camp fire with my friends, she explored the creek a lot further out than I thought, took care of younger kids, somehow latched on to the idea of building a dam, and put that into action. I didn’t suggest any of that to her. She did it all on her own.

Seeing her creating something passionately, not just consuming what’s given to her, was such a tremendous relief. I mean, look at her happy face and you will see. She is perfectly capable of finding something she wants to do. I don’t know what that’s going to be in the real life, but I just felt so confident that she will be fine. As parents, I and my wife were watering and fertilizing this little plant every day hoping that some day it will grow into a beautiful flower, whatever color that might be, always worrying that it has too few leaves, or too many leaves. But maybe parenting is more like where we think this plant will bloom into a flower, then discover that the soil turns into a clay and the clay turns into a beautiful flower vase. One giant sequence of beautiful unintended consequences.

When I look at it like that, my heart is filled with deep, deep appreciations to so many.

First and the foremost, my wife. Throughout this post, I said “we” like I played the equal part, but let’s be honest, I was mostly doing just talking and thinking. It is my wife who has been doing almost all the doing, while I was flying around the world pursuing my own passion. Unintended consequence doesn’t mean you can skip parenting. It is her 15 years of constant watering and feeding that created enough mass that allows this little plant to turn into something. And only I get to watch and appreciate that never-ending devotion. I just hope a similar lightning strikes her and relieves her from the fear and the pressure.

Then there are people of the group. They got me hooked into camping in the first place, which led to this moment, and they keep on giving. Activities to my daughter that her parents wouldn’t have thought of themselves is one but that’s not the only thing. They provide her compliments, push her further, teach her fishing, and otherwise interact with her in their own unique ways. I don’t have any real extended family, certainly not on this side of the Pacific ocean, but I just realized these people are practically my big extended family. And they don’t know what it means to me. What a wonderful bunch of folks they are.

I’m surrounded by amazing people. And it’s wonderful to be aware of that. This is a day that I’m going to remember.

Support API freedom

I was reading this article from Steve and Sacha about the API copyrightability, and found myself in a violent agreement. If you haven’t read it, I highly recommend it.

For those of you who haven’t been following the tech news, the issue at hand is Android — Google neatly side-stepped Java’s compatibility requirements by introducing a new runtime/VM and said that Android is not Java. Oracle sued Google claiming that the Java API is a copyrightable material, and that Google can’t just create a whole new implementation that’s API-compatible with Java. Oracle lost the case, but now Oracle is appealing, and they are garthering legacy vendor friends to argue that API not copyrightable is bad for economy.

But wait, surely more competition is bad for them vendors, but what about the instant gain we the developers got when Android came along, in becoming instantly productive in this entirely new platform?

Looking at the comment secion of the article, I was bit disappointed that some people saw this only as a storm in a teacup, or that this is an issue only about Java, and their favorite programming ecosystem (C#, Ruby, …) are OK. But it’s quite a contrary. If the appeal is successful, it has a broad implication on all sorts of APIs. As they say, first they come for the communists, and you think you are safe, but by the time they come to you, it might be too late!

Take Mono for example. Sure, C# and CLI are under Microsoft Community Promise. But what about the vast APIs in the .NET Framework, which is necessary for writing any meaningul application? What about all the Win32 APIs that Wine implements? Or how about Eucalyptus implementing the Amazon Web Services API? Sure, they might be in a good relationship now, but what if IBM acquires Eucalyptus and started a cloud offering of the same API?

As a developer I benefit every day from the compatibility and being able to migrate from one vendor to another without losing everything. And when I look back at PC/AT, x86 instruction sets, Java EE APIs, and so on, I truly believe that the openness is good not just for us the developers but for the broader economy as a whole.

So after reading the article, I felt like I wanted to help the cause and voice my support, but I wasn’t sure how — I’m just a developer and not a lawyer. So I created a White House petition. Not so much because I expect the White House to do something about it, but it’s a good enough neutral petition site that hopefully people feel safe enough to join. If you agree with the cause, please join the petition and help spread the words, so that our voices get heard.

JavaFX needs to be a new edition of Java

Lately, there has been a number of security vulnerabilities reported in Java. The latest one is reported just after a few days of JavaSE 7u11, which by itself a response to another vulnerability. It’s so bad to the point that people are being asked to uninstall Java (yes, just in the browser, but let’s face it, it’s way easier to just uninstall it entirely than to disable the Java plugin from all the browsers.)

I think this issue needs to be addressed at a deeper level pretty quickly, or else what’s still left of the client-side Java would be dead soon (if you think it’s already dead, then there’s nothing else to see in this post, so please move on.)

In fact, one could even argue that the entire Java platform is at risk — people (especially our users and those who are studying programming) do not understand that these issues are the sandbox breaches and therefore do not affect the server-side Java, nor embedded Java.

I think the main lesson should be that the sandbox model of Java is unmaintainable. It’s not just a few isolated bugs here and there, but it’s a structural problem.

For those of you who aren’t very familiar with Java, basically Java sandbox model is that:

  • At runtime the entirety of Java core libraries are present (the exact same code you run on the server-side, including such abilities as forking processes, making network connections to anywhere, and accessing files)
  • Code gets associated with the “code source” information to indicate if it’s trusted or not
  • Library code (like one that accesses files) checks if the caller is trusted or not, and if not certain operations aren’t allowed.

The check in step #3 is implemented in a Java code (called SecurityManager), and that’s where the problem is. All the recent vulnerabilities in Java basically involves runtime reflection to replace the effective SecurityManager instance. This forces the reflection library to be inside the protection wall built by the security layer, which in turns forces many libraries in JavaSE to be inside the wall (such as JMX, which is used for the recent exploit, as well as XML stuff, which I was personally involved in.)

At this point, the protection wall is so long and windy that it’s impossible to defend. And I think it’s really all because we decided to protect reflection.

IMHO, a much simpler way to let untrusted code run is just to enforce checks on the native side, and simply stop trusting anything that runs as Java byte code. After all, we really need to just check stuff that interacts with the outside world, and every one of those has to be done by calling into native code. There’s very few of these calls. In this way, neither JMX nor XML APIs can be exploited, because they simply become just another untrusted code (that just so happen to be available all the time.) I’m no Flash expert, but IIUC, this is how Flash security model works.

In other words, create a new edition of Java, by adding one more to the ME/SE/EE mix. It is a new runtime environment that just shares the same virtual machine and the subset of core libraries. I think this is already consistent with the way “JavaFX” is marketed. While technically speaking it’s just a fancy UI library, it’s really marketed as an RIA platform along ME/SE/EE.

With the sophisticated existing IDEs, static typing, people’s familiarity, and the formidable existing ecosystems, I think Java (VM and the byte code in particular, and to somewhat less extent the language) is still incredibly useful to the large-scale client side development. Especially so if a bit more can be added, such as a better interoperability with DOM and other HTML5 APIs. While JavaScript is getting better day by day, there are still many things Java does better.

Imagine being able to define a large application into modules, with interfaces to define boundaries, mature module systems, efficient delivery and caching mechanisms. Debugger support, multi-threading, runs outside the browser for unit testing, programming in Ruby or Groovy … you get the idea. Think of it as GWT except translation, if you may.

As an avid Java fan, here’s my plea to Oracle — I want this for my Christmas present this year.

Internet Explorer only takes string in HTML5 postMessage

Today I learned the hard way that our beloved Internet Explorer doesn’t handle HTML5 cross-domain messaging functionality (AKA postMessage) correctly.

I have done a reasonable amount of research before I invested a significant time into this, and so I knew it doesn’t postMessage across tabs and windows beforehand. This limitation was not a problem for me, because I was going to use this in IFRAME.

The problem that I discovered much too late is that IE only accepts a string as a message (see BSTR in a signature? That’s a Windows speak for passing UTF-16 strings.) This is despite the fact that the draft spec allowed any object and not just strings as early as Novemember 2010 .

And other sites like and various blog posts failed me too, as they failed to mention this limitation.

So I’m posting this here in the hope that this will prevent other people from wasting time.


Jenkins User Conference New York: my take

In one of those days, I’ll get a small enough computer whose battery won’t die in 45 minutes, but until then, my apologies for the belated Jenkins User Conference travel report.

So, as we tour around the world bringing Jenkins User Conference near you, the last stop was New York City.

I’ve kicked off the day by recaping the community activities in the past year, including Subversion 1.7 support, new UI effort, and several new extension points in the Matrix project. From my day job side, I’ve introduced BuildHive, a new addition to CloudBees DEV@cloud that builds your GitHub repositories in a few clicks, and a new version of Jenkins Enterprise by CloudBees with high-availability features.

There are two parallel tracks that went through the whole day, including talks from people who has been around in the community for a long time. Just to name a few, there’s Monty Taylor from HP discussing how OpenStack deploys Jenkins and Gerrit to ensure that the trunk never breaks, and Mike Rooney giving advices on running mission-critical Jenkins instances. Jesse Farinacci was also there, even though we couldn’t convince him to give a talk (and you can see his recap post in

But aside from those who I already knew, there were many talks from whom I call “super Jenkins admins” — those who not only deploys Jenkins for their orgs, but push it to 11, by writing custom plugins, changing the visualization, writing peripheral programs that interact with Jenkins, etc.

For JUC NY, the super Jenkins admin award has to go to Jesse and David from AtTask. There are a number of awesome things they’ve done. First, they completely took the Jenkins instance to cloud, and not only do they provision build slaves from EC2, but also provision the entire Selenium Grid and its remote agents as well. Then they implemented a custom view plugin so that developers can see which stage in the pipeline their changes are at. They’ve also integrated Jenkins to their own product, AtTask, and automatically turns test failures into tickets (and it’s smart enough to reopen the old ones when a regression occurs!)

One thing I felt as I was listening to these incredible user story talks is that it’d be nice if we have means to capture the configuration of those instances and share those with others. Kind of like how you share cooking recipes. I’m not exactly sure how to achieve it, but it should include a list of plugins, some configurations, and it’d let you import those into your instance, so that you can replicate it, play with it, and modify it easily.

And last but not least, The talk from Noah about Jenkins REST API was also very good, in that it really highlighted a part of Jenkins that’s not widely used, and it gives everyone something that they can take back to their home.

I hope this post encourages you to come to future JUCs. Please submit papers, and register from the website. I’m really looking forward to the next one in Israel, and the one after that, Tokyo, has already signed up 700 people, so it’ll be a crazy event!

Groovy SecureASTCustomizer is harmful

I was looking at Groovy DSL slides from Guillaume Laforge when I noticed about SecureASTCustomizer, which led me to what appers to be the original introduction post from Cedric.

Being able to lock Groovy execution down would enable me to use Groovy in more places, so I did a bit of experiment. But I regrettably have to conclude that this feature is practically unusable. In fact I’d argue that it is actively harmful, as it gives a programmer a false comfort.

The fundamental problem is that Groovy is a dynamic language, yet SecureASTCustomizer works by looking at Groovy AST statically. So it’s very easy for Maloney, a malicious attacker, to bypass many of the checks. For example, Cedric’s post talks about how it can let you blacklist/whitelist classes that can be imported. Well, the actual goal of the programmer is to prevent the class from getting used, and not to get them imported. And sure enough, even if I white list the importable classes to java.lang.Math, Maloney can still do Math.class.forName('some.secret.class') to get a reference to a Class, and therefore render the import restrictions pointless.

Then I thought about disabling access to the getClass() method. But this doesn’t work well either because Groovy allows 5."class" and 5["class"] to access properties. To statically prevent this, you’d have to prohibit the array access and a string literal, but that doesn’t leave much of a language!

Many other checks offered by SecureASTCustomizer are equally useless. For example, there’s receiversClassesWhiteList that’s supposed to let you restrict the methods the script can invoke by whitelisting the declaring class of the method. But once again, this is a static check! Groovy compiler doesn’t work very hard to infer types, so much so that it can’t even guess that x=="foo" is a boolean type. Therefore, if you actually try using receiver whitelisting, pretty quickly you’ll discover that you either have to allow Object as a receiver (because Groovy assigns this to every expression when it couldn’t infer the type), which will basically renders the point of whitelisting moot as you can now invoke any method by simply casting the expression to Object.

If you go the other route and disallow Object as a receiver. That will reject almost all non-trivial scripts. Or I suppose you can prohibit a method call, but that doesn’t leave much of a language, does it.

Like I said, I think this is fundamentally a futile approach. You just can’t perform any meaningful static sandboxing on a dynamic language.

Instead, what I think is more fruitful is a dynamic checking. For example, what if the compile-time AST transformation intercepts every method call and property access? That is, transform z=x.y as z=checkedGet(x,"y"), transform x.y=5 into checkedSet(x,"y",5), and finally transform,b,c) into checkedCall(o,"foo",[a,b,c]). This does make execution a whole lot slower, but I can now perform meaningful checks. And unlike Java SecurityManager, this is a lot more friendly to libraries and web applications, who cannot take over the entire JVM.

I haven’t actually put together such an AST transformer, but this doesn’t look too hard.

What do people think?

COM4J updates

It’s been a while, but I’ve posted a new version of COM4J. COM4J is a library that lets you talk to Windows COM components. Unlike similar libraries lika jacob, which makes you feel like you are working with reflection, COM4J is designed to work with type-safe annotated interfaces, which makes you feel like you are working with Java libraries. COM4J is also built on top of vtable invocation, not on IDispatch, so it can work with components without the dual interface support (boy those words bring back memories!)

I use this library in Jenkins, among other places, to provide a better native integration.

The major change in this version is that it finally has 64bit Java support. The original work was contributed in 2011, but I’ve never cut a release out of it officially. It contains a number of bug fixes, additional conversions support. The code is now on GitHub, and the website is moved to here.

Debian and Maven, a crash of culture

Tim O’Brien posted his frustration about the state of Java packaging in Debian. While I’m not affiliated with Debian nor Ubuntu, I wanted to post something in defense.

I completely understand where Tim is coming from. To the eyes of Java developers, the Java packaging in Debian looks completely Sisyphean. We got all the binaries and their dependencies captured in a machine readable form (aka POM). Can’t we just take them as-is, do a bit of metadata conversion, and make all those artifacts available to the Debian world so that we can just have a single package manager on Debian? If that’s your line of reasoning, you are in for a surprise, because Debian wouldn’t like that.

The reason they don’t do it is well summarized in the Debian Social Contract. It’s the equivalent of the U.S. Constitution for the Debian project — everything they do derive from this. Binary jars are bad for Debian because they don’t give the users the freedom to modify them and create derivative works. Debian is not just a means to let you conveniently install all the programs you need. It’s a pursuit of certain kinds of freedom.

In that sense, it’s somewhat like the “Free Software” movement. They both have some pretty strong guiding principles, and at times, for outsiders they look like they are “wasting” their efforts or being impractical. But the thing is, it’s those guiding principles that attract so many people to the effort, and that’s what keeps the project going and produce all the incredible good stuff that we use everyday. Criticizing them for their principles while you enjoy the benefits of the very same principles feel bit single-handed to me.

I think a better way forward is to write a little program that takes the source jar (which most jars in the Maven central should already have) and the POM, then generate a build script that simply compiles the source jar into the binary jar. The said program should also inspect the jar file to figure out any resource files, and treat them as source files. That way, we can machine-generate Debian source packages. Granted, not all source packages produced that way would pass the requirements of the Debian Freesoftware Guideline, but I bet substantial number of Maven artifacts are simple enough that this will be actually completely satisfactory. And then humans can concentrate on harder ones.

Anyone interested in giving that a shot?

POTD: Package renamed ASM

Today’s project of the day is a package renamed ASM library.

I previously wrote about a problem in otherwise quite useful ObjectWeb ASM library. Namely, it breaks backward compatibility in such a way that badly breaks apps/libraries that use them. In that post, I wrote about two proposals to fix the pain point. One is to include debug information, which has been fixed starting 3.x. But the other package renaming hasn’t been addressed in the last 2 years.

This has been in the back of my head, but it never came high enough until recently I had another NoSuchMethodError caused by ASM3. One of the servlet containers shipped ASM3 and it broke Jenkins that bundles ASM2. Between this and ASM4 release for JavaSE 7, which will likely gain popularity over time, I finally decided to fix this problem once and for all, in a way that everyone else can reuse.

The solution, as explained in the original post, is to put each major ASM version in its unique package name. I pakage-renamed ASM2 in org.kohsuke.asm2, ASM3 in org.kohsuke.asm3, and ASM4 in org.kohsuke.asm4. The package name only contains the major version because I trust the ASM developers to maintain compatibility between minor releases (and I believe they’ve maintained this thus far.)

These artifacts are available in org.kohsuke:asm2:2.2.3, org.kohsuke:asm3:3.3.0, and org.kohsuke:asm4:4.0 — these are packaged renamed by jarjar and I tested them somewhat to make sure it’s not downright broken.

If library A depends on asm2 and library B depends on asm3, and someone else uses both A and B, everything will work fine because asm2 and asm3 are in the different pcakages. If A depends on one version of asm3 and B depends on another version of asm3, then the transitive dependency resolution will pick up the newer version and both will work (or you end up implicitly picking up one version over another, and you don’t enjoy the latest bug fixes, but at least it won’t die with LinkageError.)

When you search “asm3″ in Maven central, you see a large number of renamed ASM3 in various projects. Hopefully that madness will stop now!

The other interesting thing about this effort is that I’ve used Gradle to package rename them. Lately I’ve been using Gradle for publishing transformed artifacts like these to a Maven repository, and I like it a lot. But more about that in another post.