Configuring Your Cucumber.js Tests with Playwright

saucectl relies on a YAML specification file to determine exactly which tests to run and how to run them. To customize saucectl to run your Cucumber.js tests with Playwright, simply modify the properties of the YAML file accordingly. This page defines each of the configuration properties specific to running Cucumber.js with Playwright tests.

Setting an Alternative Configuration File

By default, saucectl looks for the config.yml file in the .sauce folder of your project root, but you can actually specify a different file, or if you are using multiple frameworks or need to configure different sets of tests to run separately, you may choose to have multiple configuration files that you can direct saucectl to reference as necessary.

Use the following configuration at runtime to direct saucectl to use any configuration file you choose:

saucectl run -c ./path/to/{config-file}.yml

YAML Required

While you can use multiple files of different names or locations to specify your configurations, each file must be a *.yml and follow the saucectl syntax. Our IDE Integrations (e.g., Visual Studio Code) can help you out by validating the YAML files and provide handy suggestions, so make sure to check them out!

Example Configuration

examples/cucumber/.sauce/config.yml

loading...

View on GitHub

Each of the properties supported for running Cucumber.js with Playwright tests through saucectl is defined below.

`apiVersion`

| REQUIRED | STRING |

Identifies the version of the underlying configuration schema. At this time, v1alpha is the only supported value.

apiVersion: v1alpha

`kind`

| REQUIRED | STRING/ENUM |

Specifies which framework is associated with the automation tests configured in this specification.

kind: playwright-cucumberjs

`showConsoleLog`

| OPTIONAL | BOOLEAN |

Controls whether the contents of console.log are always shown in the local output of saucectl. By default (false), console.log is only shown for failed suites.

showConsoleLog: true

`defaults`

| OPTIONAL | OBJECT |

Specifies any default settings for the project.

defaults:
  timeout: 15m

`timeout`

| OPTIONAL | DURATION |

Instructs how long saucectl should wait for the suite to complete, overriding the default project timeout setting of 30 minutes.

When the suite reaches the timeout limit, its status is set to '?' in the CLI. This does not reflect the actual status of the job in the Sauce Labs web UI or API.

note

Setting 0 reverts to the value set in defaults.

defaults:
  timeout: 15m

`sauce`

| OPTIONAL | OBJECT |

The parent property containing all settings related to how tests are run and identified in the Sauce Labs platform.

sauce:
  region: eu-central-1
  metadata:
    tags:
      - e2e
      - release team
      - other tag
    build: Release $CI_COMMIT_SHORT_SHA
  concurrency: 10

`region`

| OPTIONAL | STRING/ENUM |

Specifies through which Sauce Labs data center tests will run. Valid values are: us-west-1 or eu-central-1.

note

If you do not specify a region in your config file, you must set it when running your command with the --region flag.

sauce:
  region: eu-central-1

`metadata`

| OPTIONAL | OBJECT |

The set of properties that allows you to provide additional information about your project that helps you distinguish it in the various environments in which it is used and reviewed, and also helps you apply filters to easily isolate tests based on metrics that are meaningful to you, as shown in the following example:

sauce:
  metadata:
    build: RC 10.4.a
    tags:
      - e2e
      - release team
      - beta
      - featurex

`concurrency`

| OPTIONAL | INTEGER |

Sets the maximum number of suites to execute at the same time. If the test defines more suites than the max, excess suites are queued and run in order as each suite completes.

caution

For tests running on Sauce, set this value to equal or less than your Sauce concurrency allowance, as setting a higher value may result in jobs dropped by the server.

sauce:
  concurrency: 5

Alternatively, you can override the file setting at runtime by setting the concurrency flag as an inline parameter of the saucectl run command:

saucectl run --ccy 5

`retries`

| OPTIONAL | INTEGER |

Sets the number of times to retry a failed suite. For more settings, you can refer to passThreshold.

sauce:
  retries: 1

Alternatively, you can override the file setting at runtime by setting the retries flag as an inline parameter of the saucectl run command:

saucectl run --retries 1

`tunnel`

| OPTIONAL | OBJECT |

saucectl supports using Sauce Connect to establish a secure connection with Sauce Labs. To do so, launch a tunnel; then provide the name and owner (if applicable) in this property.

sauce:
  tunnel:
    name: your_tunnel_name
    owner: tunnel_owner_username

caution

Only certain HTTP(S) ports are proxied by the tunnel.

`name`

| OPTIONAL | STRING |

Identifies an active Sauce Connect tunnel to use for secure connectivity to the Sauce Labs cloud.

note

This property replaces the former id property, which is deprecated.

sauce:
  tunnel:
    name: your_tunnel_name

`owner`

| OPTIONAL | STRING |

Identifies the Sauce Labs user who created the specified tunnel, which is required if the user running the tests did not create the tunnel.

note

This property replaces the former parent property, which is deprecated.

sauce:
  tunnel:
    name: your_tunnel_name
    owner: tunnel_owner_username

`timeout`

| OPTIONAL | DURATION |

How long to wait for the specified tunnel to be ready. Supports duration values like '10s', '30m' etc. (default: 30s)

sauce:
  tunnel:
    name: your_tunnel_name
    timeout: 30s

`visibility`

| OPTIONAL | STRING |

Sets the visibility level of test results for suites run on Sauce Labs. If unspecified or empty, team visibility will be applied. Valid values are:

public: Accessible to everyone.
public restricted: Share your job's results page and video, but keeps the logs only for you.
share: Only accessible to people with a valid link.
team: (Default) Only accessible to people under the same root account as you.
private: Only you (the owner) will be able to view assets and test results page.

sauce:
  visibility: private

`launchOrder`

| OPTIONAL | STRING |

Specifies the execution order for your test suites. When set to fail rate, test suites with the highest failure rate will execute first. If unspecified, test suites will execute in the order in which they are written in the configuration file.

sauce:
  launchOrder: fail rate

`env`

| OPTIONAL | OBJECT |

A property containing one or more environment variables that are global for all tests suites in this configuration. Values set in this global property will overwrite values set for the same environment variables set at the suite level.

env:
  hello: world
  my_var: $MY_VAR # You can also pass through existing environment variables through parameter expansion

note

Environment variables set with the saucectl --env flag will overwrite those specified in the sauce config file.

The order of precedence is as follows: --env flag > root-level environment variables > suite-level environment variables.

`rootDir`

| REQUIRED | OBJECT |

The directory of files that need to be bundled and uploaded for the tests to run. Ignores what is specified in .sauceignore. See Tailoring Your Test File Bundle for more details. The following examples show the different relative options for setting this value.

rootDir: "./" # Use the current directory

rootDir: "packages/subpackage" # Some other package from within a monorepo

caution

Only the files contained within rootDir will be available during the tests. Any reference to a file that is not included in rootDir will make the tests fail.

`npm`

| OPTIONAL | OBJECT |

A parent property specifying the configuration details for any npm dependencies. Packages listed are installed in the environment prior to your tests executing.

npm:
  strictSSL: true
  registry: https://registry.npmjs.org
  registries:
    - url: https://registry.npmjs.org
  dependencies:
    - "@cucumber/cucumber"
    - "@saucelabs/cucumber-reporter"
    - "typescript"
    - "ts-node"

`registry`

| OPTIONAL | STRING |

note

This setting is supported up to Playwright 1.35.1. For newer versions, use registries.

Specifies the location of the npm registry source. If the registry source is a private address and you are running tests on Sauce Cloud, you can provide access to the registry source using Sauce Connect.

npm:
  registry: https://registry.npmjs.org

`registries`

| OPTIONAL | ARRAY |

Specifies the location of the npm registry, scope, and credentials. Only one scopeless registry is allowed. If the registry is inside a private network, you must establish a tunnel using Sauce Connect.

npm:
  registries:
    - url: https://registry.npmjs.org
    - url: https://private.registry.company.org
      scope: "@company"
      authToken: secretToken
      auth: base64SecretToken
      username: myUsername
      password: myPassword
      email: myEmail 

`url`

Specifies the URL of the npm registry.

| REQUIRED | STRING |

npm:
  registries:
    - url: https://registry.npmjs.org

`scope`

Specifies which scope is associated with this registry. See Associating a scope with a registry.

| OPTIONAL | STRING |

npm:
  registries:
    - url: https://registry.npmjs.org
      scope: "@company"

`authToken`

Specifies the authentication token to be used with this registry.

| OPTIONAL | STRING |

npm:
  registries:
    - url: https://registry.npmjs.org
      authToken: secretToken

`auth`

Specifies the Base64-encoded authentication string for the registry entry.

| OPTIONAL | STRING |

npm:
  registries:
    - url: https://registry.npmjs.org
      auth: base64SecretToken

`username`

Specifies the username for authentication with the registry.

| OPTIONAL | STRING |

npm:
  registries:
    - url: https://registry.npmjs.org
      username: myName

`password`

Specifies the password for authentication with the registry.

| OPTIONAL | STRING |

npm:
  registries:
    - url: https://registry.npmjs.org
      password: myPassword

`email`

Specifies the email associated with the registry account.

| OPTIONAL | STRING |

npm:
  registries:
    - url: https://registry.npmjs.org
      email: myEmail

`dependencies`

| OPTIONAL | ARRAY |

Specifies any npm packages that are required to run tests and should, therefore, be included in the bundle. The dependencies specified here have to be already installed in the local node_modules folder. These dependencies, along with any related transitive dependencies, are then included in the bundle that is uploaded to Sauce Labs.

In order to run Cucumber.js tests with Playwright, you must to install the following required packages locally, and then add the dependencies to the configuration file.

npm:
  dependencies:
    - "@cucumber/cucumber" # Cucumber official suggested package
    - "@saucelabs/cucumber-reporter" # Sauce Labs report plugin. Generates a test cases report for display on the Sauce Labs UI.
    - "typescript" # TypeScript support
    - "ts-node" # TypeScript support

To use this feature, make sure that node_modules is not ignored via .sauceignore.

note

saucectl doesn't support running Cucumber.js tests with Playwright via installing Cucumber related packages on the fly.

`packages`

| OPTIONAL | OBJECT |

Specifies any npm packages that are required to run tests and should, therefore, be installed on the Sauce Labs VM. See Including Node Dependencies.

npm:
  packages:
    lodash: "4.17.20"
    "@babel/preset-typescript": "7.12"

caution

Do not use dependencies and packages at the same time.

`strictSSL`

| OPTIONAL | BOOLEAN |

Instructs npm to perform SSL key validation when making requests to the registry via HTTPS (true) or not (false). Defaults to npm's strict-ssl value if not set. See more here.

npm:
  strictSSL: false
  package:
    "lodash": "4.17.20"

`reporters`

| OPTIONAL | OBJECT |

Configures additional reporting capabilities provided by saucectl.

reporters:
  junit:
    enabled: true
    filename: saucectl-report.xml

`junit`

| OPTIONAL | OBJECT |

The JUnit reporter gathers JUnit reports from all jobs and combines them into a single report.

reporters:
  junit:
    enabled: true
    filename: saucectl-report.xml

`json`

| OPTIONAL | OBJECT |

The JSON reporter gathers test results from all jobs and combines them into a single report.

reporters:
  json:
    enabled: true
    filename: saucectl-report.json
    webhookURL: https://my-webhook-url

`enabled`

| OPTIONAL | BOOLEAN |

Toggles the reporter on/off.

reporters:
  json:
    enabled: true

`webhookURL`

| OPTIONAL | STRING |

Specifies the webhook URL. When saucectl test is finished, it'll send an HTTP POST with a JSON payload to the configured webhook URL.

reporters:
  json:
    enabled: true
    webhookURL: https://my-webhook-url

`filename`

| OPTIONAL | STRING |

Specifies the report filename. Defaults to "saucectl-report.json".

reporters:
  json:
    enabled: true
    filename: my-saucectl-report.json

`artifacts`

| OPTIONAL | OBJECT |

Specifies how to manage test artifacts, such as logs, videos, and screenshots.

artifacts:
  cleanup: true
  download:
    when: always
    match:
      - junit.xml
    directory: ./artifacts/

`cleanup`

| OPTIONAL | BOOLEAN |

When set to true, all contents of the specified download directory are cleared before any new artifacts from the current test are downloaded.

artifacts:
  cleanup: true

`retain`

| OPTIONAL | OBJECT |

Define directories to archive and retain as a test asset at the end of a test run. Archived test assets can be downloaded automatically using the download configuration, via the REST API, or through the test details page.

artifacts:
  retain:
    source-directory: destination-archive.zip
  download:
    when: always
    match:
      - destination-archive.zip
    directory: ./artifacts/

note

The source and destination will be relative to the rootDir defined in your configuration.

note

The destination archive must have a .zip file extension.

`download`

| OPTIONAL | OBJECT |

Specifies the settings related to downloading artifacts from tests run by saucectl.

artifacts:
  cleanup: true
  download:
    when: always
    match:
      - junit.xml
    directory: ./artifacts/

`when`

| OPTIONAL | STRING |

Specifies when and under what circumstances to download artifacts. Valid values are:

always: Always download artifacts.
never: Never download artifacts.
pass: Download artifacts for passing suites only.
fail: Download artifacts for failed suites only.

artifacts:
  cleanup: true
  download:
    when: always

`match`

| OPTIONAL | STRING/ARRAY |

Specifies which artifacts to download based on whether they match the name or file type pattern provided. Supports the wildcard character * (use quotes for best parsing results with wildcard).

artifacts:
  cleanup: true
  download:
  match:
    - junit.xml
    - "*.log"

`directory`

| OPTIONAL | STRING |

Specifies the path to the folder location in which to download artifacts. A separate subdirectory is generated in this location for each suite for which artifacts are downloaded. The name of the subdirectory will match the suite name. If a directory with the same name already exists, the new one will be suffixed by a serial number.

artifacts:
  cleanup: true
  download:
    directory: ./artifacts/

`allAttempts`

| OPTIONAL | BOOLEAN |

If you have your tests configured with retries, you can set this option to true to download artifacts for every attempt. Otherwise, only artifacts of the last attempt will be downloaded.

artifacts:
  download:
    match:
      - console.log
    when: always
    allAttempts: true
    directory: ./artifacts/

`playwright`

| REQUIRED | OBJECT |

The parent property containing the details specific to the Playwright project.

playwright:
  version: 1.39.0

`version`

| REQUIRED | STRING |

The version of Playwright that is compatible with the tests defined in this file. See Supported Testing Platforms for the list of Playwright versions supported by saucectl and their compatible test platforms.

playwright:
  version: 1.39.0

tip

You can also define a path to your package.json. This will make saucectl use the same Playwright version that's defined in your projects devDependencies or dependencies map.

The path to your package.json file will be relative to the rootDir of your configuration.

`suites`

| REQUIRED | OBJECT |

The set of properties providing details about the test suites to run. May contain multiple suite definitions. See the full example config for an illustration of multiple suite definitions.

`name`

| REQUIRED | STRING |

The name of the test suite, which will be reflected in the results and related artifacts.

suites:
  - name: "saucy test"

`browserName`

| OPTIONAL | STRING |

Sets the browser name for the test suite. saucectl passes browserName as the environment variable $BROWSER_NAME.

The user is responsible for launching the correct browser via Cucumber.js. Using a different browser, than specified, results in mismatched job metadata.

suites:
  - name: "saucy test"
    browserName: "chromium"

`env`

| OPTIONAL | OBJECT |

A property containing one or more environment variables that may be referenced in the tests for this suite. Expanded environment variables are supported. Values set here will be overwritten by values set in the global env property.

suites:
  - name: "saucy test"
    env:
      hello: world
      my_var: $MY_VAR

`platformName`

| OPTIONAL | STRING |

A specific operating system and version on which to run the specified browser and test suite. Defaults to a platform that is supported by saucectl for the chosen browser.

suites:
  - name: "saucy test"
    platformName: "Windows 11"

`screenResolution`

| OPTIONAL | STRING |

Specifies a browser window screen resolution, which may be useful if you are attempting to simulate a browser on a particular device type. See Test Configurations for a list of available resolution values.

suites:
  - name: "saucy test"
    screenResolution: "1920x1080"

`shard`

| OPTIONAL | STRING |

When sharding is configured, saucectl automatically splits the tests (e.g., by spec or concurrency) so that they can easily run in parallel. For sharding by concurrency, saucectl splits test files into several groups (the number of groups is determined by the concurrency setting). Each group will then run as an individual job.

Selectable values: spec to shard by spec file, concurrency to shard by concurrency. Remove this field or leave it empty "" for no sharding.

suites:
  - name: 'I am sharded'
    shard: spec

tip

To split tests in the most efficient way possible, use:

spec when the number of specs is less than the configured concurrency.
concurrency when the number of specs is larger than the configured concurrency.

`shardTagsEnabled`

| OPTIONAL | BOOLEAN |

When sharding is configured and the suite is configured to filter scenarios by tags, it is possible for feature files to be allocated to VMs only to be skipped if the feature file doesn't contain any scenarios matching the specified tags.

With shardTagsEnabled enabled, saucectl will filter out feature files that do not contain scenarios matching the given tags. This will prevent wasted VM allocations.

suites:
  - name: A shard with tags example suite
    shard: spec
    shardTagsEnabled: true
    options:
      tags:
        - "@smoke and not @flakey"

`timeout`

| OPTIONAL | DURATION |

Instructs how long saucectl should wait for the suite to complete, potentially overriding the default project timeout setting.

When the suite reaches the timeout limit, its status is set to '?' in the CLI. This does not reflect the actual status of the job in the Sauce Labs web UI or API.

note

Setting 0 reverts to the value set in defaults.

suites:
  - name: "saucy test"
    timeout: 15m

`preExec`

| OPTIONAL | STRING/ARRAY |

Specifies which commands needs to be executed before the tests are actually started. The commands are executed from the root directory of your project.

note

There is a 300-second limit for all preExec commands to complete.

suites:
  - name: "saucy test"
    preExec:
      - node ./scripts/pre-execution-script.js

`passThreshold`

| OPTIONAL | INTEGER |

Specifies the minimum number of successful attempts for a suite to be considered as passed. It should be used along with retries.

note

For example, setting retries to 3 and passThreshold to 2. The max attempt would be 4 times. If the test passed twice, it'd stop and be marked as passed. Otherwise, it'd be marked as failed.

sauce:
  retries: 3
suites:
  - name: My Saucy Test
    passThreshold: 2

`smartRetry`

| OPTIONAL | OBJECT |

Specifies the retry strategy to apply for that suite. Requires retries to be >= 1.

sauce:
  retries: 3
suites:
  - name: My Saucy Test
    smartRetry:
      failedOnly: true

`failedOnly`

| OPTIONAL | BOOLEAN |

When set to true, only the spec files that failed during the previous attempt are retried.

suites:
  - name: My Saucy Test
    smartRetry:
      failedOnly: true

`options`

| REQUIRED | OBJECT |

Provides details related to the Cucumber configuration that are relevant for this test suite.

suites:
  - name: My Cucumber Test
    browserName: chromium
    options:
      paths:
        - "features/**/*.feature"
      require:
        - "features/support/*.js"
      format:
        - "json:my-cucumber.json"

`config`

| OPTIONAL | STRING |

Specifies the path to the Cucumber configuration file. See the Cucumber.js Configuration documentation for more information.

suites:
  - name: My Cucumber Test
    options:
      config: "my_cucumber_config.js"

`name`

| OPTIONAL | STRING |

Specifies with regular expression matching which Cucumber scenarios to run. See the Cucumber.js Filtering documentation for more information.

suites:
  - name: My Cucumber Test
    options:
      name: ".*My Cucumber Scenario"

`paths`

| REQUIRED | ARRAY |

Paths to feature files. See the Cucumber.js Configuration documentation for more information. You can use glob pattern to indicate all files that match a specific value, such as a file name, type, or directory.

suites:
  - name: My Cucumber Test
    options:
      paths:
        - "features/**/*.feature"

`excludedTestFiles`

| OPTIONAL | ARRAY |

Excludes test files to skip the tests. You can use glob pattern to indicate all files that match a specific value, such as a file name, type, or directory.

suites:
  - name: My Cucumber Test
    options:
      excludedTestFiles: ["features/failed/**/*.feature"]

`backtrace`

| OPTIONAL | BOOLEAN |

Specifies whether to show the full backtrace for errors.

suites:
  - name: My Cucumber Test
    options:
      backtrace: true

`require`

| OPTIONAL | ARRAY |

Paths to your support code for CommonJS. See the Cucumber.js Configuration documentation for more information.

suites:
  - name: My Cucumber Test
    options:
      require:
        - "features/support/*.js"

`import`

| OPTIONAL | ARRAY |

Paths to your support code for ESM. See the Cucumber.js ES Modules (experimental) documentation for more information.

suites:
  - name: "saucy test"
    options:
      import:
        - "features/support/*.js"

`tags`

| OPTIONAL | ARRAY |

Tag expression to filter which Cucumber scenarios run. See the Cucumber.js Filtering documentation for more information.

suites:
  - name: My Cucumber Test
    options:
      tags:
        - "@smoke"
        - "@e2e"

`format`

| OPTIONAL | ARRAY |

Name/path and (optionally) output file path of each formatter to use. See the Cucumber.js Formatters documentation for more information.

suites:
  - name: My Cucumber Test
    options:
      format:
        - "json:my-cucumber.json"

`formatOptions`

| OPTIONAL | OBJECT |

Options to provide to formatters. See the Cucumber.js Formatters documentation for more information.

suites:
  - name: My Cucumber Test
    options:
      formatOptions:
        someOption: true

Setting an Alternative Configuration File​

Example Configuration​

apiVersion​

kind​

showConsoleLog​

defaults​

timeout​

sauce​

region​

metadata​

concurrency​

retries​

tunnel​

name​

owner​

timeout​

visibility​

launchOrder​

env​

rootDir​

npm​

registry​

registries​

url​

scope​

authToken​

auth​

username​

password​

email​

dependencies​

packages​

strictSSL​

reporters​

junit​

json​

enabled​

webhookURL​

filename​

artifacts​

cleanup​

retain​

download​

when​

match​

directory​

allAttempts​

playwright​

version​

suites​

name​

browserName​

env​

platformName​

screenResolution​

shard​

shardTagsEnabled​

timeout​

preExec​

passThreshold​

smartRetry​

failedOnly​

options​

config​

name​

paths​

excludedTestFiles​

backtrace​

require​

import​

tags​

format​

formatOptions​

Setting an Alternative Configuration File

Example Configuration

`apiVersion`

`kind`

`showConsoleLog`

`defaults`

`timeout`

`sauce`

`region`

`metadata`

`concurrency`

`retries`

`tunnel`

`name`

`owner`

`timeout`

`visibility`

`launchOrder`

`env`

`rootDir`

`npm`

`registry`

`registries`

`url`

`scope`

`authToken`

`auth`

`username`

`password`

`email`

`dependencies`

`packages`

`strictSSL`

`reporters`

`junit`

`json`

`enabled`

`webhookURL`

`filename`

`artifacts`

`cleanup`

`retain`

`download`

`when`

`match`

`directory`

`allAttempts`

`playwright`

`version`

`suites`

`name`

`browserName`

`env`

`platformName`

`screenResolution`

`shard`

`shardTagsEnabled`

`timeout`

`preExec`

`passThreshold`

`smartRetry`

`failedOnly`

`options`

`config`

`name`

`paths`

`excludedTestFiles`

`backtrace`

`require`

`import`

`tags`

`format`

`formatOptions`