Attaching files to JUnit tests

Despite the fact that it is the de-facto standard of test reports in any programming languages (perhaps except .NET), JUnit test report format has a number of problems. One is that the format isn’t explicitly defined (and I’ll discuss this in a separate post), but another problem, which I’m going to dedicate this post for, is the lack of attachment support.

It is often very conveniet to be able to attach arbitrary files to a test report. Imagine Selenium tests capturing screenshots. Or JavaEE tests that deploy webapps, then you want to capture server log files. Or if you are doing UI automation testing, how about a video recording that your screen automation framework has produced?

Today, doing this mostly requires that you (as a test author) write some files somewhere, then print out that file name to stdout/stderr. This works for humans who are looking at the output, but not for CI servers like Jenkins. So I hereby propose a bit of convention to decorate this current practice, to make those files discoverable by Jenkins.

For this, I’ve improved the JUnit attachments plugin to recognize the following format. It has to occupy a whole line.

[[ATTACHMENT|<absolute file name>]]
[[ATTACHMENT|<absolute file name>|{ ... additional metadata in JSON ... }]]

The additional metadata isn’t currently used, but I intended it to describe what the attached file means. For example, if your test always attach a couple of log files, it’d be useful to describe which file is which, so that CI servers or test report tools can display them as such. Or metadata for human readable display name would be useful, as these attachment file names are often cryptic just to make them unique.

When you run these tests from within Jenkins, these files are then picked up and stored by Jenkins, and the test report page will include them as links.

Ideally, the test report format should be expanded to cover things like this, but unfortunately I think that’d require too much collaboration between too many people to the point that it’s unrealistic — if we are to do that, test frameworks like JUnit first needs to offer this as API methods for listeners, then the test drivers like Ant/Maven needs to be expanded to honor those when they produce test reports. Then finally we can improve the CI servers.

I’ve been patiently waiting for that to happen for long time, but it’s just not happening. So instead, I’m taking the matter into my own hands, and came up with this convention.

Convention like this is useful only if enough people uses it. So I hope you’ll like this. If you think this convention can be improved, please let me know.

11 thoughts on “Attaching files to JUnit tests”

  1. Hi,
    I use this plugin for junit reports with the stderr convention and it works fine !
    But, i have also some projects using testng and unfortunately, it doesn’t work …. I try to customize the listener to format the testng console output as junit one…failed…
    The only solution is to use the folder convention ? Your opinion ?
    Thx
    Benoît coming to Paris Jenkins Conference

  2. Hi Kohsuke!

    The main difference between JUnit, TestNG and other xUnit protocols and TAP and SubUnit is that TAP and SubUnit are extensible. Perl default format is TAP, Node.js is slowly adopting it as well, PHP and a lot of other languages support it too (including Java).

    With TAP you can add YAMLish data to a test result. I’ve wrote about TAP and TestNG some time ago [1][2], and about the difference between TAP and SubUnit (after a discussion with some Jenkins users in Jenkins dev-list) [3].

    In the next weeks I’ll update tap4j parser to support other Perl tools such as ‘prove’ and subtests. Recently I’ve received many requests in TAP plug-in and tap4j, and discovered that these requests were coming mainly from Perl and node.js developers. I was aware that Perl developers would adore this plug-in, but had no idea node.js was using TAP. You can see Testem CI, for instance, that supports TAP + Jenkins through the TAP plug-in.

    I believe that if JUnit and TestNG adopted TAP as an alternative output format, we would have a lot of new node.js, Perl, Python (node can output TAP, as well as Jython+tap4j), Shell and other technologies coming to Jenkins :)

    I’ve been waiting for the same happening with test protocols, but I think it the unit frameworks provided an extensible output format (could be TAP, SubUnit, or a third too), changes in Maven and other tools would be minor.

    [1] http://www.kinoshita.eti.br/2011/03/20/discussing-the-flexibility-of-tap-to-cover-information-generated-by-testng/
    [2] http://www.kinoshita.eti.br/2011/03/10/getting-the-maximum-from-testng-with-tap/
    [3] http://www.kinoshita.eti.br/2011/06/04/a-comparison-of-tap-test-anything-protocol-and-subunit/

  3. hi kohsuke, i’m trying to associate an attachment to a test case. so far, i have only been successful with associating an attachment to a test class, which may contain multiple test cases. perhaps i don’t quite understand the junit attachments plugin.

  4. Great idea, but it doesn’t work. If it ever work it’s so fragile that sane people should stay away! Tested on Windows with latest version of Jenkins and this plug-ins.

  5. This is exactly what I need. But, unfortunately, the plugin doesn’t work on my build server :( This is what I have done:

    * I installed the plugin on my Jenkins
    * In my test case I added code like this:
    System.out.println(“[[ATTACHMENT|” + file.getAbsolutePath() + “]]”);

    But nothing happens. No overview of the collected files anywhere. Do I have to activate this plugin anyhow?

  6. @Jan
    You need to configure it on the job itself: Add a “Additional test report features” as a “Post-build action”, tick “Publish test attachments”.

    Sadly, I’ve done so but still haven’t gotten it to work. I’m using federated build servers, maybe that might be the reason. If anyone got this to work I’m keen to learn!

  7. I’m having the same problem. I added the “Additional test report feature” to the “Publish JUnit test result report” post-build action.
    I’m using a node.js script that prints the “[[ATTACHMENT|/path/to/file]]” to stdout, but after the job no attachments are shown on the test result page.

    Any news about this?

  8. I am in the exact situation as Fabian – running mocha tests with webdriverio, with a node.js script that does a console log of “[[ATTACHMENT|/path/to/file]]” right at the point before failing tests. I have a ‘Publish JUnit test result report’ post build action with the ‘Publish test attachments’ Additional test report feature. In the Failed test all I see is the Error Message and Stacktrace.

  9. I was able to resolve the issue. With mocha/webdriverio, and perhaps with other node.js solutions, the key is to write to a system-out element of the junit report, not simply print to console output.

  10. Finally! A successful Smoke test displaying all 40 screen shots!
    On every job the ‘Retain long standard output/error’ option needs to be checked under PublishJUnit test result report plugin configuration. This will prevent the output from being truncated which was preventing the screen shot attachments from being attached. This will use much more space on Jenkins, so storing less past runs, or less verbose logs may be required.

  11. I managed to get this to work from our NUnit tests by printing [[Attachment|absolute/path/to/file]] to the TestContext System Out element. I add new lines before and after, since the [[ATTACHMENT..]] needs to be on its own line.

    TestContext.Out.WriteLine();
    TestContext.Out.WriteLine($"[[ATTACHMENT|{fileName}]]");
    TestContext.Out.WriteLine();

    Since I use the NUnit ConsoleRunner, I transform the NUnit test result to JUnit using nunit3-junit.xsl

    nunit3-console.exe YourTestAssembly.dll --result=junit-results.xml;transform=nunit3-junit.xslt

    In Jenkins, I publish the JUnit report using the JUnit Plugin. Since I am using pipeline, the publish call will look something like this:

    junit allowEmptyResults: true, keepLongStdio: true, testDataPublishers: [[$class: 'AttachmentPublisher']], testResults: 'testresults/*.xml'

    This corresponds to the JUnit settings “Retain long standard output/error” and “Additional test report features” with “Publish test attachments” available from the plugin created by Kohsuke.

    This was done in an older Jenkins version, 2.44.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>