Support API freedom

April 5th, 2013

I was reading this article from Steve and Sacha about the API copyrightability, and found myself in a violent agreement. If you haven’t read it, I highly recommend it.

For those of you who haven’t been following the tech news, the issue at hand is Android — Google neatly side-stepped Java’s compatibility requirements by introducing a new runtime/VM and said that Android is not Java. Oracle sued Google claiming that the Java API is a copyrightable material, and that Google can’t just create a whole new implementation that’s API-compatible with Java. Oracle lost the case, but now Oracle is appealing, and they are garthering legacy vendor friends to argue that API not copyrightable is bad for economy.

But wait, surely more competition is bad for them vendors, but what about the instant gain we the developers got when Android came along, in becoming instantly productive in this entirely new platform?

Looking at the comment secion of the article, I was bit disappointed that some people saw this only as a storm in a teacup, or that this is an issue only about Java, and their favorite programming ecosystem (C#, Ruby, …) are OK. But it’s quite a contrary. If the appeal is successful, it has a broad implication on all sorts of APIs. As they say, first they come for the communists, and you think you are safe, but by the time they come to you, it might be too late!

Take Mono for example. Sure, C# and CLI are under Microsoft Community Promise. But what about the vast APIs in the .NET Framework, which is necessary for writing any meaningul application? What about all the Win32 APIs that Wine implements? Or how about Eucalyptus implementing the Amazon Web Services API? Sure, they might be in a good relationship now, but what if IBM acquires Eucalyptus and started a cloud offering of the same API?

As a developer I benefit every day from the compatibility and being able to migrate from one vendor to another without losing everything. And when I look back at PC/AT, x86 instruction sets, Java EE APIs, and so on, I truly believe that the openness is good not just for us the developers but for the broader economy as a whole.

So after reading the article, I felt like I wanted to help the cause and voice my support, but I wasn’t sure how — I’m just a developer and not a lawyer. So I created a White House petition. Not so much because I expect the White House to do something about it, but it’s a good enough neutral petition site that hopefully people feel safe enough to join. If you agree with the cause, please join the petition and help spread the words, so that our voices get heard.

Uncategorized , ,

Jenkins commit activities in 2012

January 25th, 2013

Stephen is having fun with Git and R, and I saw another person creating a similar comparison (although I’m not sure where the credit should go on that one.)

Both charts are great, but I also noticed that their Y-axis aren’t consistent. So if people aren’t careful they might not see Jenkins as favorably as it actually is. So here is the graph, comparing # of commits in the year 2012. They both only includes the core and in the master branch:

On average, we have 41.2 commits per week, compared to 9.7 on the other side. If you want the raw data, here is the Excel file.

hudson, jenkins

JavaFX needs to be a new edition of Java

January 20th, 2013

Lately, there has been a number of security vulnerabilities reported in Java. The latest one is reported just after a few days of JavaSE 7u11, which by itself a response to another vulnerability. It’s so bad to the point that people are being asked to uninstall Java (yes, just in the browser, but let’s face it, it’s way easier to just uninstall it entirely than to disable the Java plugin from all the browsers.)

I think this issue needs to be addressed at a deeper level pretty quickly, or else what’s still left of the client-side Java would be dead soon (if you think it’s already dead, then there’s nothing else to see in this post, so please move on.)

In fact, one could even argue that the entire Java platform is at risk — people (especially our users and those who are studying programming) do not understand that these issues are the sandbox breaches and therefore do not affect the server-side Java, nor embedded Java.

I think the main lesson should be that the sandbox model of Java is unmaintainable. It’s not just a few isolated bugs here and there, but it’s a structural problem.

For those of you who aren’t very familiar with Java, basically Java sandbox model is that:

  • At runtime the entirety of Java core libraries are present (the exact same code you run on the server-side, including such abilities as forking processes, making network connections to anywhere, and accessing files)
  • Code gets associated with the “code source” information to indicate if it’s trusted or not
  • Library code (like one that accesses files) checks if the caller is trusted or not, and if not certain operations aren’t allowed.

The check in step #3 is implemented in a Java code (called SecurityManager), and that’s where the problem is. All the recent vulnerabilities in Java basically involves runtime reflection to replace the effective SecurityManager instance. This forces the reflection library to be inside the protection wall built by the security layer, which in turns forces many libraries in JavaSE to be inside the wall (such as JMX, which is used for the recent exploit, as well as XML stuff, which I was personally involved in.)

At this point, the protection wall is so long and windy that it’s impossible to defend. And I think it’s really all because we decided to protect reflection.

IMHO, a much simpler way to let untrusted code run is just to enforce checks on the native side, and simply stop trusting anything that runs as Java byte code. After all, we really need to just check stuff that interacts with the outside world, and every one of those has to be done by calling into native code. There’s very few of these calls. In this way, neither JMX nor XML APIs can be exploited, because they simply become just another untrusted code (that just so happen to be available all the time.) I’m no Flash expert, but IIUC, this is how Flash security model works.

In other words, create a new edition of Java, by adding one more to the ME/SE/EE mix. It is a new runtime environment that just shares the same virtual machine and the subset of core libraries. I think this is already consistent with the way “JavaFX” is marketed. While technically speaking it’s just a fancy UI library, it’s really marketed as an RIA platform along ME/SE/EE.

With the sophisticated existing IDEs, static typing, people’s familiarity, and the formidable existing ecosystems, I think Java (VM and the byte code in particular, and to somewhat less extent the language) is still incredibly useful to the large-scale client side development. Especially so if a bit more can be added, such as a better interoperability with DOM and other HTML5 APIs. While JavaScript is getting better day by day, there are still many things Java does better.

Imagine being able to define a large application into modules, with interfaces to define boundaries, mature module systems, efficient delivery and caching mechanisms. Debugger support, multi-threading, runs outside the browser for unit testing, programming in Ruby or Groovy … you get the idea. Think of it as GWT except translation, if you may.

As an avid Java fan, here’s my plea to Oracle — I want this for my Christmas present this year.

Uncategorized

On the road for the rest of the month

January 10th, 2013

I’m really excited to kick off my 2013 with a tour around the world for Jenkins.

The first stop will be in Tel Aviv, where I’ll be doing training with AlphaCSP. This has sold out, but AlphaCSP will be delivering this training again in the future. I’ll then head to London for the Jenkins User Event in London, with James Nord (of the m2 release plugin fame) and Andrew Phillips from XebiaLabs. This one has also sold out, but I’ll be recording the talks and posting them online. After tha, I’m in Munich to deliver another Jenkins training with ObjectBay. Again the same deal here — it’s sold out but they’ll be scheduling repeats later.

I’ll then be back to my home for a few days, then off to Seoul for the first-ever Jenkins meet-up in Korea. I’m really excited about this, and I’m trying to kick-start a local community, which is invaluable in this region of the world. If you are in the area, please come join us, and get to know other people in the Jenkins community.

Then I head to Tokyo to an user meet-up (no registration page is up yet, please check back our website later), and a couple more trainings (this one has more seats available.)

From there I’m flying to Brussels for FOSDEM. They gave me a keynote slot (yay!), and Tyler is organizing the testing and automation devroom. A number of Jenkins people will be there, and we’ll have a table for two days. This will be exciting!

And in my last leg of the journey, I’ll be at JFokus in Stockholm. They kindly gave me a long 3 and half hour slot to talk about Jenkins in depth, aside from more typical 50 mins talking slot. I’m looking forward to this new format.

And when I finally get back to home, I suspect I need some down time to recover.

If you are in the area nearby, and grab some stickers or want to say hello, please let me know! And my apologies in advance for the delay in responding e-mails and other disruptions.

jenkins , , , ,

The other side of forking and pull requests

January 4th, 2013

Charles Nutter of JRuby fame had this tweet yesterday:

And this touches on something I’ve been thinking for a long time.

My experience mostly comes from the Jenkins project, which is one of the open commit policy projects. Everyone gets to be a committer just by asking, and that grants them access to all the plugins. This policy is deep in our culture that we even have an IRC bot that the project people use to add new committers.

This is not necessarily a good idea for every project, but it certainly served us well, especially back in the days when we had the entire source code in the Subversion repository. During that time, for people to make changes to the plugins, they really needed to be a committer — maintaining a custom patch set on top of an external Subversion repository is a royal pain in the back side. The open commit policy means anyone interested in committing can join the project without a fuss, and once they become a contributor and start commiting code, they feel like being a part of the project, and that feeds fresh blood into the Jenkins developer community.

While I totally believe in the technological superiority of Git as a version control system over Subversion, in my view, the fork and pull-request driven world of Git hasn’t been a complete win for the Jenkins project.

Forking and pull requests do encourage contributions by eliminating certain barriers to start hacking on someone else’s project. Whereas it was quite painful to maintin patches externally to someone else’s source code in Subversion, Git makes this trivial with branching and occasional rebases or merges. This part is a good thing.

On the other hand, this ease of maintaining your own versions reduces the incentive to bring your changes back to the community, to some extent. After all, you’ve already scratched your itch and it works on your machine. Why bother trying to push the code back and go through the trouble of writing tests and documenting the change, convincing others about the design, and arguing why that change was needed to begin with. You can see this in a number of forks people created that never turns into as pull requests, and I’ve done this a number of times myself.

An even bigger problem in my mind is that by allowing people to contribute by just sending pull requests, it encourages the “throw the stuff over the wall” way of contributing. Sometimes people can do stuff without telling us anything upfront, and you as a receiver would be hard-pressed to turn it down, given that someone spent so much effort in it. Sometimes when we request additional changes, they never come back. It makes the collaboration and communication after the fact, not before the coding. And it also makes the pathway to the committer bit unclear. How do we convert people from the side sending pull requests to the side receiving pull requests? If we have to ask them one by one it won’t scale.

The situation gets worse on plugins, where there may not be an active maintainer at the moment. We’d like those who are sending us pull requests to take over the maintainership, and use their best judgement to evaluate the changes. But when GitHub makes them feel like the proper way to contribute is by sending a pull request, and if your pull requests do not get any attention, I’m not sure how they are supposed to realize that the plugin needs a new owner. Back in the days of Subversion, the repository alone was enough to be a center of gravity of the project. It attracts new developers (so long as we let them be, aka open commit policy) and it’s almost self sustaining. Now it takes more than just the repository.

In any case, I just wanted to point out that the impliations of forking and pull requests are bit more nuanced.

P.S. I’ve been long meaning to write a bot that patrols for such unattended pull requests, and encourage them to become committers, but haven’t had a chance to do so yet. I still intend to do this, and we’ll see how it goes.

jenkins , , ,

POTD: StopForumSpam API for Java

December 22nd, 2012

There’s an on-going spam problem in Jenkins wiki. We have capture, but either that was broken, or more likely, spamming is done manually.

One of the suggestions was to integrate StopForumSpam service. To that end, I wrote a simple Java client library for their API. The usage would be something like this:

    for (Answer a : new StopForumSpam().build().ip(ip).email(email).query()) {
        if (a.isAppears()) {
            LOGGER.warning("Rejecting, likely spam: "+a);
            throw new UserError("Due to the spam problem, we need additional verification for your sign-up request. Please contact jenkinsci-dev@googlegroups.com");
        }
    }

My hats off to them for providing a great service, and here is hoping that this will help make the dent. We’ll see.

potd

In São Paulo from this weekend

November 26th, 2012

I’ll be flying to Sao Paulo this week to attend Jenkins meet-up (Saturday) and JavaOne Latin America (next Tuesday and onward).

The last time I visited Brazil was a few years ago, but thanks to Mauricio Leal, whom I tagged around with for a JUG tour around Brazil, it was really a blast. This time around I don’t have the benefit of the local guy to the degree I had before, but I’m really looking forward to it.

The major pre-travel issue was that the Brazilian government run out of the visa stickers. But I made it just in time (or more precisely I hopefully will be — I’ll be picking up my passport Wednesday and flying Thursday. Phew!)

The highlight of the trip will be the first-eve Jenkins meet up in São Paulo — I’ve been a big fan of local communities where there exists a language barrier. I’m from Japan, so I’ve seen it the first hand the value of local communities would bring to the table. The speaker/session line up is amazing, the event is free, and there’ll be a social at the end, so if you are in the area and has been interested in meeting with fellow Jenkins developers and users, please RSVP.

But if you can’t come in person, at least please consider joinining the Brazilian Portaguese mailing list of Jenkins users, which we are launching.

I don’t really have any plans on Sunday and Monday — any sight-seeing suggestions and/or interest of meeting up and chatting about stuff would be more than welcome. Leave the comment here, or see the right for my e-mail address.

jenkins , , ,

POTD: submit a patch to Jenkins, and let him test it for you

October 8th, 2012

Here’s my 2nd after-JavaOne “project of the day” Jenkins plugin. This has been in the back of my mind for quite some time, but it took this gentleman to grill me on this feature during JavaOne for me to finally put it together — so thank YOU for doing that. although I didn’t catch your name.

The plugin is called the “patch parameter” plugin. This plugin lets you submit a patch when scheduling a build, and that gets applied to the checked out source tree before a build would commence. Any failure to apply a patch will result in a build failure.

This plugin can be used for a “pre-tested commit” workflow. You can work on a change locally, have the diff tested on the server, then if you are satisfied, you can commit it. I can imagine this would be also an interesting building block for integration with code review tools.

I should also note that we have the Subversion Merge plugin for a different approach to a similar problem, and distributed VCS can generally do this better (for example I do one for Git in CloudBees Validated Merge plugin.)

jenkins, potd , ,

POTD: iOS device connector plugin (cont’d)

October 7th, 2012

Today, I wrapped up the project I started Friday and released iOS device connector plugin.

In addition to listing all iOS devices, this plugin lets you deploy IPA files from anywhere (in Jenkins build via a build step or outside via CLI) to any of the connected iOS devices.

There are still some loose-ends that I’d like to get some feedback on:

  • You should be able to select a device without electing a specific UDID, and I’m curious what kind of criteria people needs here. Do you need to be able to say “deploy my app on iPad2 with 16GB memory”, or is “deploy my app to an available iPad” suffice?
  • For this to be useful for automated test execution. I think we need to be able to launch the app (and I assume the tests embedded in the app itself will run and shuts itself down in the end?) How do people do it?
  • Don’t we need to inject the coordinates of the actual device the app was deployed to, so that your build can do more stuff with it?
  • What are other interesting operations to devices. File access? Screenshot?

Anyway, the plugin is already released. I hope you find it useful.

jenkins, potd

POTD: iOS device detection in Jenkins

October 5th, 2012

I was talking to my colleague Mark Prichard about mobile development with Jenkins, and I came up with this idea.

If you are doing iOS app testing with real devices, you need to tether the device with a computer so that you can push the app-to-be-tested to the device. In a local development environment, you’d naturally do this by hooking up a device to your laptop.

But in the CI environment, we can do better. Imagine if you have a Jenkins master connected to a dozen Mac minis, where each Mac mini is then connected to various iOS devices. There’s nothing stopping Jenkins from listing up all the devices connected to all the slaves. Then based on this information, Jenkins can lease devices to builds.

Say someone wants to run a test somewhere. Jenkins can find an unused device and push it, then inject the coordinate of that device to the build. Or how about the axis in a matrix project that allows you to say “I need this test to run with iPhone 3GS,4,5 and iPad 2,3″ then Jenkins will run 5 runs of the tests in parallel?

To that end, as a post-JavaOne fun project, I developed a plugin that lists up all the iOS devices connected to Jenkins.

And just to show that we can list up various properties of this device:

I should be able to combine this with something like fruitstrap so that you can deploy apps from anywhere to any device connected to your Jenkins build farm.

What do you think? Shouldn’t this be interesting? Any thoughts?

jenkins, potd , ,