Debian and Maven, a crash of culture

Tim O’Brien posted his frustration about the state of Java packaging in Debian. While I’m not affiliated with Debian nor Ubuntu, I wanted to post something in defense.

I completely understand where Tim is coming from. To the eyes of Java developers, the Java packaging in Debian looks completely Sisyphean. We got all the binaries and their dependencies captured in a machine readable form (aka POM). Can’t we just take them as-is, do a bit of metadata conversion, and make all those artifacts available to the Debian world so that we can just have a single package manager on Debian? If that’s your line of reasoning, you are in for a surprise, because Debian wouldn’t like that.

The reason they don’t do it is well summarized in the Debian Social Contract. It’s the equivalent of the U.S. Constitution for the Debian project — everything they do derive from this. Binary jars are bad for Debian because they don’t give the users the freedom to modify them and create derivative works. Debian is not just a means to let you conveniently install all the programs you need. It’s a pursuit of certain kinds of freedom.

In that sense, it’s somewhat like the “Free Software” movement. They both have some pretty strong guiding principles, and at times, for outsiders they look like they are “wasting” their efforts or being impractical. But the thing is, it’s those guiding principles that attract so many people to the effort, and that’s what keeps the project going and produce all the incredible good stuff that we use everyday. Criticizing them for their principles while you enjoy the benefits of the very same principles feel bit single-handed to me.

I think a better way forward is to write a little program that takes the source jar (which most jars in the Maven central should already have) and the POM, then generate a build script that simply compiles the source jar into the binary jar. The said program should also inspect the jar file to figure out any resource files, and treat them as source files. That way, we can machine-generate Debian source packages. Granted, not all source packages produced that way would pass the requirements of the Debian Freesoftware Guideline, but I bet substantial number of Maven artifacts are simple enough that this will be actually completely satisfactory. And then humans can concentrate on harder ones.

Anyone interested in giving that a shot?

Come join us on “Selenium, Jenkins, Robots, Oh My!” tomorrow

I’ll be speaking tomorrow at San Francisco Selenium Meetup about Jenkins & Selenium — mainly recent improvements in the Selenium plugin, as well as several other new plugins relevant in the combination of Jenkins and Selenium, complete with a demo. I’ve got a couple of pet-peeves against the Selenium project, so I’m going to pitch them there to see the reaction, too.

I’ve told that Jason Huggins from Sauce Lab is going to pick up where he left off in the last Jenkins User Conference to talk about his robot, and Theo Cincotta from Eventbrite will give the case study of how Eventbrite uses Jenkins & Selenium together internally, so the whole thing should be a great mix of fun & useful topics, all packed in a Wednesday night from 6:30pm to 8:00pm, with beer and food.

The event is free, but you do need to RSVP, instead of the usual RSVP in the SF Selenium meetup group page at (which currently says 209 people coming, when the EventBrite RSVP page says the capacity is 100 people — so I need to check with the organizers to make sure they know what they are expecting…)

Attaching files to JUnit tests

Despite the fact that it is the de-facto standard of test reports in any programming languages (perhaps except .NET), JUnit test report format has a number of problems. One is that the format isn’t explicitly defined (and I’ll discuss this in a separate post), but another problem, which I’m going to dedicate this post for, is the lack of attachment support.

It is often very conveniet to be able to attach arbitrary files to a test report. Imagine Selenium tests capturing screenshots. Or JavaEE tests that deploy webapps, then you want to capture server log files. Or if you are doing UI automation testing, how about a video recording that your screen automation framework has produced?

Today, doing this mostly requires that you (as a test author) write some files somewhere, then print out that file name to stdout/stderr. This works for humans who are looking at the output, but not for CI servers like Jenkins. So I hereby propose a bit of convention to decorate this current practice, to make those files discoverable by Jenkins.

For this, I’ve improved the JUnit attachments plugin to recognize the following format. It has to occupy a whole line.

[[ATTACHMENT|<absolute file name>]]
[[ATTACHMENT|<absolute file name>|{ ... additional metadata in JSON ... }]]

The additional metadata isn’t currently used, but I intended it to describe what the attached file means. For example, if your test always attach a couple of log files, it’d be useful to describe which file is which, so that CI servers or test report tools can display them as such. Or metadata for human readable display name would be useful, as these attachment file names are often cryptic just to make them unique.

When you run these tests from within Jenkins, these files are then picked up and stored by Jenkins, and the test report page will include them as links.

Ideally, the test report format should be expanded to cover things like this, but unfortunately I think that’d require too much collaboration between too many people to the point that it’s unrealistic — if we are to do that, test frameworks like JUnit first needs to offer this as API methods for listeners, then the test drivers like Ant/Maven needs to be expanded to honor those when they produce test reports. Then finally we can improve the CI servers.

I’ve been patiently waiting for that to happen for long time, but it’s just not happening. So instead, I’m taking the matter into my own hands, and came up with this convention.

Convention like this is useful only if enough people uses it. So I hope you’ll like this. If you think this convention can be improved, please let me know.

POTD: Package renamed ASM

Today’s project of the day is a package renamed ASM library.

I previously wrote about a problem in otherwise quite useful ObjectWeb ASM library. Namely, it breaks backward compatibility in such a way that badly breaks apps/libraries that use them. In that post, I wrote about two proposals to fix the pain point. One is to include debug information, which has been fixed starting 3.x. But the other package renaming hasn’t been addressed in the last 2 years.

This has been in the back of my head, but it never came high enough until recently I had another NoSuchMethodError caused by ASM3. One of the servlet containers shipped ASM3 and it broke Jenkins that bundles ASM2. Between this and ASM4 release for JavaSE 7, which will likely gain popularity over time, I finally decided to fix this problem once and for all, in a way that everyone else can reuse.

The solution, as explained in the original post, is to put each major ASM version in its unique package name. I pakage-renamed ASM2 in org.kohsuke.asm2, ASM3 in org.kohsuke.asm3, and ASM4 in org.kohsuke.asm4. The package name only contains the major version because I trust the ASM developers to maintain compatibility between minor releases (and I believe they’ve maintained this thus far.)

These artifacts are available in org.kohsuke:asm2:2.2.3, org.kohsuke:asm3:3.3.0, and org.kohsuke:asm4:4.0 — these are packaged renamed by jarjar and I tested them somewhat to make sure it’s not downright broken.

If library A depends on asm2 and library B depends on asm3, and someone else uses both A and B, everything will work fine because asm2 and asm3 are in the different pcakages. If A depends on one version of asm3 and B depends on another version of asm3, then the transitive dependency resolution will pick up the newer version and both will work (or you end up implicitly picking up one version over another, and you don’t enjoy the latest bug fixes, but at least it won’t die with LinkageError.)

When you search “asm3″ in Maven central, you see a large number of renamed ASM3 in various projects. Hopefully that madness will stop now!

The other interesting thing about this effort is that I’ve used Gradle to package rename them. Lately I’ve been using Gradle for publishing transformed artifacts like these to a Maven repository, and I like it a lot. But more about that in another post.

POTD: checking package name for Windows reserved words

Today’s project of the day is a little tool that checks problems in your package names that can bite you on Windows.

For backward compatibility with MS-DOS (which had ended its life more than 15 years ago!), Windows doesn’t allow you to create files/directories of certain names, such as “AUX” or “PRN” (see Microsoft Knowledge Base.)

This can bite those of us who develop on Unix — when you accidentally create a file or a directory of this name (say in the package name), it’ll be all right for you, but your colleagues who use Windows will not be able to build them. Worse yet, if none of your colleagues use Windows either, this problem can get unnoticed until you ship/test the code.

So I wrote a little tool that proactively checks for this problem. It’s an annotation processor that kicks in automatically, so you just need to add it in your classpath when you compile your source files. If you use a build system that can handle transitive dependencies (such as Gradle or Maven), you just need to use it in one of your libraries, and all the code that uses your library also benefits from this check.

My hats off to Microsoft for their rigorous commitment to the backward compatibility, and just one more reason why you should be always testing your Java program on multiple platforms…