2016. április 19.

Exploring Behat ep. 1: formatting test results

Ábrahám Zsófia
Cloud Engineer

As a growing company with a strong Drupalist department, we have reached a point when continuous integration and automated testing is necessary to sustain pace, and given the characteristics of Drupal, behavior-driven testing with Behat is a logical candidate. To make this happen, we have to explore the undocumented territories of Behat, and we are presenting our findings along the way.


Test output configuration options

When working in a continuous integration ecosystem, test output has to be formatted in a readable and standardized way. Behat is said to offer several formatting options, so I’m running behat --help for starters. It spits out the following options related to test output:

  1. -f, --format=FORMAT How to format tests output. pretty is default.
  2.     Available formats are:
  3.         progress: Prints one character per step.
  4.         pretty: Prints the feature as is.
  5.     You can use multiple formats at the same time. (multiple values allowed)
  6. -o, --out=OUT Write format output to a file/directory instead of STDOUT (output_path).
  7.     You can also provide different outputs to multiple formats. (multiple values allowed)
  8. --format-settings=FORMAT-SETTINGS
  9.     Set formatters parameters using json object.
  10.     Keys are parameter names, values are values. (multiple values allowed)
  11. --colors Force ANSI color in the output. 
  12.     By default color support is guessed based on your platform and the output if not specified.
  13. --no-colors Force no ANSI color in the output.

Colored output

Now --color and --no-color is pretty straightforward, but the other three are worded quite sparingly.


For --format, the allowed values are there but the phrase “multiple values are allowed” is ambiguous, running behat --format='pretty,progress' throws an error. Multiple formats should be specified as separate --format options like behat --format=pretty --format=progress.

Outputting results to file

Since --out can be set to different values for each format, the order of the formats is also significant.


Note that when there are multiple formats and one output target specified, all of the formats’ output goes to the same place. So to log one format to stdout and one to a file, you have to write

  1. behat \
  2.     --format=pretty \
  3.     --out=./reports/behat.pretty.log \ 
  4.     --format=progress \
  5.     --out=/dev/stdout

A common use case of using multiple formats for the same test would be a CI setting where one format output is printed to the developer in a human-readable format and the other gets logged somewhere in a machine readable format, ie. a json or xml. The good news is that the next version of behat (3.1.0) is already in rc2 state and supports jUnit xml format natively.

Format settings

Color scheme

My primary use case for formatting behat output was to change the color scheme because some color combinations were unreadable on my dark terminal. Behat help says that --format-settings, the primary candidate for altering color schema, accepts a json format but there is no hint about the structure, and the docs show no link to color settings. Fortunately there was a documentation for behat 2.5 color settings, so I had a good starting point, though, it only talks about setting globals in behat.yml. The default color settings are as follows:

  1. default:
  2.   formatters:
  3.     pretty:
  4.       output_styles:
  5.         passed:        ['green']
  6.         passed_param:  ['green',  null,  ['bold']]
  7.         undefined:     ['yellow', null,  ['bold', 'underscore']]
  8.         pending:       ['yellow', null,  ['underscore']]
  9.         pending_param: ['yellow', null,  ['bold', 'underscore']]
  10.         failed:        ['white', 'red']
  11.         failed_param:  ['white',  'red', ['bold']]
  12.         skipped:       ['cyan']
  13.         skipped_param: ['cyan',   null,  ['bold']]
  14.         comment:       ['default', null]
  15.         tag:           ['cyan']

Behat is using Symfony’s OutputStyleFormatter class here, so every element config is an array of foreground color, background color and an array of text style options.

“Available foreground and background colors are: black, red, green, yellow, blue, magenta, cyan and white. And available options are: bold, underscore, blink, reverse (enables the "reverse video" mode where the background and foreground colors are swapped) and conceal (sets the foreground color to transparent, making the typed text invisible - although it can be selected and copied; this option is commonly used when asking the user to type sensitive information).” (source: OutputStyleFormatter docs)

The docs suggest that the --format-settings option expects the configuration below “formatter” level in a JSON format, so in order to set comments to a readable white and tags to a unique magenta, I’m running behat --format-settings='{"output_styles":{"tag":["magenta"],"comment":["white"]}}'.


Other settings

There are three other format options that seem to work, all boolean switches.

  1. defaults:
  2.   formatters:
  3.     pretty:
  4.       paths: true     # Display path where step is defined.
  5.       snippets: true  # Display example snippets for missing steps.
  6.       multiline: true # Display multiline arguments.
  7.       output_styles: {}

CAVEAT: providing multiple --format-settings values doesn‘t behave as expected, since it seems to only count the last valid value and display every formatter with the same settings despite having a help line on accepting multiple options (and calling a foreach in the code, so in this case, it’s probably a bug (as of version 3.0.15)).

In the following example, the first case with one --format-settings value behaves as expected (passing steps turn magenta), but when it’s run with two formats turned on, it produces identical results for both formatters, with yellow failures and green passes, so the first --format-settings value is completely ignored. (When given one format and multiple --format-settings value, the same override happens.)


That’s it. I wouldn’t say Behat test formatting is a very complex beast with endless capabilities, but it certainly has some nice convenience features. Looking forward to more advanced options and a seamless CI experience in the future.

Next time we’re going to explore Behat scenario selectors.

Related posts

2016. június 8.

In our previous post on the topic, we used output formatters to determine how Behat displays test results. Now we continue with exploring our possibilities on what tests to run together with Behat’s scenario selectors.