Home > jenkins > Attaching files to JUnit tests

Attaching files to JUnit tests

March 13th, 2012

Despite the fact that it is the de-facto standard of test reports in any programming languages (perhaps except .NET), JUnit test report format has a number of problems. One is that the format isn’t explicitly defined (and I’ll discuss this in a separate post), but another problem, which I’m going to dedicate this post for, is the lack of attachment support.

It is often very conveniet to be able to attach arbitrary files to a test report. Imagine Selenium tests capturing screenshots. Or JavaEE tests that deploy webapps, then you want to capture server log files. Or if you are doing UI automation testing, how about a video recording that your screen automation framework has produced?

Today, doing this mostly requires that you (as a test author) write some files somewhere, then print out that file name to stdout/stderr. This works for humans who are looking at the output, but not for CI servers like Jenkins. So I hereby propose a bit of convention to decorate this current practice, to make those files discoverable by Jenkins.

For this, I’ve improved the JUnit attachments plugin to recognize the following format. It has to occupy a whole line.

[[ATTACHMENT|<absolute file name>]]
[[ATTACHMENT|<absolute file name>|{ ... additional metadata in JSON ... }]]

The additional metadata isn’t currently used, but I intended it to describe what the attached file means. For example, if your test always attach a couple of log files, it’d be useful to describe which file is which, so that CI servers or test report tools can display them as such. Or metadata for human readable display name would be useful, as these attachment file names are often cryptic just to make them unique.

When you run these tests from within Jenkins, these files are then picked up and stored by Jenkins, and the test report page will include them as links.

Ideally, the test report format should be expanded to cover things like this, but unfortunately I think that’d require too much collaboration between too many people to the point that it’s unrealistic — if we are to do that, test frameworks like JUnit first needs to offer this as API methods for listeners, then the test drivers like Ant/Maven needs to be expanded to honor those when they produce test reports. Then finally we can improve the CI servers.

I’ve been patiently waiting for that to happen for long time, but it’s just not happening. So instead, I’m taking the matter into my own hands, and came up with this convention.

Convention like this is useful only if enough people uses it. So I hope you’ll like this. If you think this convention can be improved, please let me know.

jenkins , ,

  1. Benoit
    March 27th, 2012 at 11:00 | #1

    Hi,
    I use this plugin for junit reports with the stderr convention and it works fine !
    But, i have also some projects using testng and unfortunately, it doesn’t work …. I try to customize the listener to format the testng console output as junit one…failed…
    The only solution is to use the folder convention ? Your opinion ?
    Thx
    Benoît coming to Paris Jenkins Conference

  2. April 5th, 2012 at 17:19 | #2

    Hi Kohsuke!

    The main difference between JUnit, TestNG and other xUnit protocols and TAP and SubUnit is that TAP and SubUnit are extensible. Perl default format is TAP, Node.js is slowly adopting it as well, PHP and a lot of other languages support it too (including Java).

    With TAP you can add YAMLish data to a test result. I’ve wrote about TAP and TestNG some time ago [1][2], and about the difference between TAP and SubUnit (after a discussion with some Jenkins users in Jenkins dev-list) [3].

    In the next weeks I’ll update tap4j parser to support other Perl tools such as ‘prove’ and subtests. Recently I’ve received many requests in TAP plug-in and tap4j, and discovered that these requests were coming mainly from Perl and node.js developers. I was aware that Perl developers would adore this plug-in, but had no idea node.js was using TAP. You can see Testem CI, for instance, that supports TAP + Jenkins through the TAP plug-in.

    I believe that if JUnit and TestNG adopted TAP as an alternative output format, we would have a lot of new node.js, Perl, Python (node can output TAP, as well as Jython+tap4j), Shell and other technologies coming to Jenkins :)

    I’ve been waiting for the same happening with test protocols, but I think it the unit frameworks provided an extensible output format (could be TAP, SubUnit, or a third too), changes in Maven and other tools would be minor.

    [1] http://www.kinoshita.eti.br/2011/03/20/discussing-the-flexibility-of-tap-to-cover-information-generated-by-testng/
    [2] http://www.kinoshita.eti.br/2011/03/10/getting-the-maximum-from-testng-with-tap/
    [3] http://www.kinoshita.eti.br/2011/06/04/a-comparison-of-tap-test-anything-protocol-and-subunit/

  3. eitan
    April 11th, 2012 at 05:35 | #3

    hi kohsuke, i’m trying to associate an attachment to a test case. so far, i have only been successful with associating an attachment to a test class, which may contain multiple test cases. perhaps i don’t quite understand the junit attachments plugin.

  4. Sebastien Tardif
    August 17th, 2012 at 08:29 | #4

    Great idea, but it doesn’t work. If it ever work it’s so fragile that sane people should stay away! Tested on Windows with latest version of Jenkins and this plug-ins.

  5. Jan
    August 2nd, 2013 at 06:26 | #5

    This is exactly what I need. But, unfortunately, the plugin doesn’t work on my build server :( This is what I have done:

    * I installed the plugin on my Jenkins
    * In my test case I added code like this:
    System.out.println(“[[ATTACHMENT|" + file.getAbsolutePath() + "]]”);

    But nothing happens. No overview of the collected files anywhere. Do I have to activate this plugin anyhow?

  6. Johan Rydström
    January 23rd, 2014 at 04:54 | #6

    @Jan
    You need to configure it on the job itself: Add a “Additional test report features” as a “Post-build action”, tick “Publish test attachments”.

    Sadly, I’ve done so but still haven’t gotten it to work. I’m using federated build servers, maybe that might be the reason. If anyone got this to work I’m keen to learn!

  1. No trackbacks yet.