Despite the fact that it is the de-facto standard of test reports in any programming languages (perhaps except .NET), JUnit test report format has a number of problems. One is that the format isn’t explicitly defined (and I’ll discuss this in a separate post), but another problem, which I’m going to dedicate this post for, is the lack of attachment support.
It is often very conveniet to be able to attach arbitrary files to a test report. Imagine Selenium tests capturing screenshots. Or JavaEE tests that deploy webapps, then you want to capture server log files. Or if you are doing UI automation testing, how about a video recording that your screen automation framework has produced?
Today, doing this mostly requires that you (as a test author) write some files somewhere, then print out that file name to stdout/stderr. This works for humans who are looking at the output, but not for CI servers like Jenkins. So I hereby propose a bit of convention to decorate this current practice, to make those files discoverable by Jenkins.
For this, I’ve improved the JUnit attachments plugin to recognize the following format. It has to occupy a whole line.
[[ATTACHMENT|<i><absolute file name></i>]]
[[ATTACHMENT|<i><absolute file name></i>|{ ... <i>additional metadata in JSON</i> ... }]]
The additional metadata isn’t currently used, but I intended it to describe what the attached file means. For example, if your test always attach a couple of log files, it’d be useful to describe which file is which, so that CI servers or test report tools can display them as such. Or metadata for human readable display name would be useful, as these attachment file names are often cryptic just to make them unique.
When you run these tests from within Jenkins, these files are then picked up and stored by Jenkins, and the test report page will include them as links.
Ideally, the test report format should be expanded to cover things like this, but unfortunately I think that’d require too much collaboration between too many people to the point that it’s unrealistic — if we are to do that, test frameworks like JUnit first needs to offer this as API methods for listeners, then the test drivers like Ant/Maven needs to be expanded to honor those when they produce test reports. Then finally we can improve the CI servers.
I’ve been patiently waiting for that to happen for long time, but it’s just not happening. So instead, I’m taking the matter into my own hands, and came up with this convention.
Convention like this is useful only if enough people uses it. So I hope you’ll like this. If you think this convention can be improved, please let me know.