June 8, 2016

Drupal

Exploring Behat Ep. 2: Behat Scenario Selectors

Ábrahám Zsófia

Cloud Engineer

In our previous post on the topic, we used output formatters to determine how Behat displays test results. Now we continue with exploring our possibilities on what tests to run together with Behat’s scenario selectors.

Running Behat with default settings executes all the tests found in the features directory. This behavior is a reasonable default, but there are a bunch of use cases when one only wants to run a subset of them. Examples:

Quickly run a scenario or feature while working on it
Test groups requiring different configuration, or that run frequently together (like grouping tests by runtime/complexity or required contexts, or to run tests on multiple servers)
Running a specific test from a generalized test collection.
For example, we have tests for common web server configurations and want to run some of them on a site.

CLI options

Let’s see what Behat offers for choosing the scenarios we want to run. First I created a couple of feature files and scenarios.

Then I looked at the command line help for hints:

$ ./bin/behat --help
Usage:
 behat [options] [--] [<paths>]

Arguments:
  paths      Optional path(s) to execute. Could be:
               - a dir (features/)
               - a feature (*.feature)
               - a scenario at specific line (*.feature:10).
               - all scenarios at or after a specific line (*.feature:10-*).
               - all scenarios at a line within a specific range (*.feature:10-20).
               - a scenarios list file (*.scenarios).
Options:
--name=NAME   Only executeCall the feature elements which match part
                of the given name or regex. (multiple values allowed)
--tags=TAGS   Only executeCall the features or scenarios with tags
                matching tag filter expression. (multiple values allowed)
--role=ROLE   Only executeCall the features with actor role matching

--name runs scenarios whose Feature or Scenario descriptions contain the given value. Providing multiple values narrows the search to those scenarios that match all the criteria.

--tags runs scenarios that are tagged with the given value. Unlike --name, --tags does not match tag names containing the value, so you cannot use, for example, a tag name prefix to run multiple tags dynamically. Also, according to the old behat documentation, tags support "&&" (and), "," (or) and "~" (not) operators in the value string.

--role runs features that belong to a given user role. Gherkin 4 can guess the user role from the feature description if it contains the line “As a(n) …”.

Running a file from CLI

We can tell behat explicitly to run one specific feature file by providing them as an argument. Despite the CLI help stating the opposite, it accepts only one path at a time and supports no wildcards (Github/Behat#834).

behat features/one.feature

Running either behat -- PATH1 PATH2 or behat PATH1 PATH2 yields an error.

You can select a portion of the file by providing a line number selector to the CLI. For example:

behat /home/me/devel/behat-test/features/other.feature:32 
behat /home/me/devel/behat-test/features/other.feature:32-40

Behat will only execute the scenarios whose title lines fall in this interval. (Unfortunately the line number selection only works with absolute paths when using the Drupal extension. [see Github/Behat#731])

Scenarios file

If we specify a path file ending in ".scenarios", behat will iterate over its lines and try to interpret them as a feature path. For example, we have a "user.scenarios" file with contents

features/user-create.feature
features/user-edit.feature

Then running behat user.scenarios executes the scenarios in both features.

Suites

CLI options are handy for exploration or development, but get clumsy when more detailed and complex options are required. It is recommended to collect tests that are frequently run together in suites. Suites are high-level test groups defined in behat.yml with completely separate configuration.

An example behat.yml:

```
default:
```
```
  suites:
```
```
    user_crud:
```
```
      paths:
```

        - '%paths.base%/features/user-create.feature'

        - '%paths.base%/features/user-misc.feature:42'

```
      filters:
```
```
        name: '...'
```
```
    slow_js:
```
```
      filters:
```
```
        tags: '@slow&&@js'
```

Running behat --suite=user_crud executes all scenarios in features/user-create.feature and features/user-misc.feature. Running behat --suite=slow_js searches across all feature files and executes any scenario tagged with @slow and @js.

Suites are well-documented on behat.org so I’m not covering them in detail here.

These are the most important test selection features in Behat, and those having any unexpected behavior. I personally miss some more dynamic mechanisms like regexp support for selectors or tags, but for most use cases it has a sufficient kit to work with.

April 4, 2018