Skip to main content

Sauce Labs with Jenkins

The OnDemand plugin allows you to easily manage your Sauce Labs testing from Jenkins, one of the most popular continuous integration platforms used in software development.

What You'll Learn​

  • How to install and configure the Sauce OnDemand Plugin for Jenkins
  • How to configure Sauce Connect to enable testing on private networks
  • How to run parallel tests in Jenkins
  • How to set up reporting between Sauce Labs and Jenkins
  • How to implement the OnDemand plugin into your Jenkins pipeline

What You'll Need​

Installing the OnDemand Plugin​

Install the Sauce OnDemand plugin from your Jenkins Administration page.

  1. From your Jenkins Dashboard, select Manage Jenkins, then Manage Plugins.
  2. Select the Available tab and choose Sauce OnDemand Plugin from the list.
  3. Click Download now and install after restart.
  4. In the plugin installation process window, select the Restart Jenkins when installation is complete and no jobs are running checkbox.
note

The plugin file is fairly large, so download may take several minutes.

Creating Your Sauce Labs Credentials​

Once you have installed the plugin and Jenkins has restarted, add your Sauce Labs credentials as environment variables so you can reference them globally for all your jobs rather than hard coding them into each test.

  1. From your Jenkins dashboard, select Manage Jenkins, then Manage Credentials.
  2. You can optionally create a unique Sauce Labs domain by hovering over the Jenkins store to reveal the context menu icon, then clicking Add domain to define the new domain in which your credentials will apply.
  3. Hover over the applicable domain and click Add Credentials from the context menu.
  4. In the Kind field, use the pull-down menu to select Sauce Labs.
  5. Provide the information in the form relevant to your Sauce Labs account:
    • Scope: Choose Global (recommended) or System based on level of access these credentials will provide in your projects.
    • Username: Enter the USERNAME value from your Sauce Labs account profile.
    • Access Key: Enter the ACCESS KEY value from your Sauce Labs account profile.
    • Sauce Data Center: Choose the Sauce Labs data center through which you run your tests.
    • ID: Enter a unique identifier for these credentials in your Jenkins environment, or leave this field blank to allow Jenkins to generate a random ID.
    • Description: Provide a brief, meaningful label for these credentials.
  6. Click OK.
  7. Back in your Jenkins dashboard, select an applicable project to apply your credentials.
  8. Choose Configure from the project menu.
  9. Select the Sauce Labs Options tab to jump to the relevant settings.
  10. Click the Credentials field and choose your credentials ID from the list.
  11. Click Save.
  12. Repeat for any additional projects that manage Sauce Labs tests.

Setting Credentials for EU Data Center Pipeline Builds​

Use the process outlined in steps 7-12 above to apply EU credentials for non-pipeline builds, but for pipeline builds in the EU data center, set the credential ID you created in Jenkins in your Pipelines Block, as shown in the following example.

stage('Test') {
    sauce('sauceuser-EU') {
        sauceconnect(useGeneratedTunnelIdentifier: true, verboseLogging: true) {
            sh 'mvn test'
        }
    }
}

Configuring Sauce Labs and Sauce Connect Settings in Jenkins​

You can manage many of the plugin settings from within the Jenkins dashboard to ensure your tests run according to your needs. The plugin is bundled with the latest version of Sauce Connect. When you enable it, you can run your Sauce Labs tests in environments that are not publicly accessible, like your local network or behind a firewall.

Some plugin options are set globally for all your Jenkins projects and some options are specific to individual projects.

note

When options can be set at both levels, project settings override global settings.

Configure Global Sauce Settings​

  1. From your Jenkins dashboard, select Manage Jenkins then Configure System.
  2. Scroll down to the Sauce Support section.
  3. Configure the optional settings as needed, based on the descriptions in the following table.
SettingDescription
Disable sending build usage stats to Sauce LabsOmit build usage information when transmitting test results data to Sauce Labs.
Override Sauce Connect PathSpecify a local path into which the Sauce Connect binary compatible with your operating system will be extracted. This value will override the default $home directory.
NOTE: Always run Sauce Connect on the same network as the site or app under test, but the same machine is not required.
Sauce Connect OptionsThe list of command line arguments to apply each time Sauce Connect Proxy is launched for any project. For example:
-i myTunnel -l ./jenkins_scp_log
Sauce Connect Max RetriesMaximum number of times Jenkins should attempt to launch a Sauce Connect tunnel before retuning a failure.
Sauce Connect Retry Wait Time in SecondsThe amount of time Jenkins should wait before retrying a failed Sauce Connect launch attempt.
Selenium Environment Variable PrefixA value that will be automatically added to the front of any Jenkins environment variable set by the Sauce OnDemand plugin.
  1. Click Save.

Configure Sauce Settings for a Project​

  1. From your Jenkins dashboard, select the project you wish to configure.
  2. Select Configure from the project menu.
  3. Select the Sauce Labs Options tab to jump to the relevant settings.
  4. Configure the optional settings as needed, based on the descriptions in the following table.
note

Project specific settings will always override the global value for the same setting.

SettingDescription
Enable Sauce ConnectLaunches a new Sauce Connect tunnel whenever Jenkins starts a build for this project and sets environment variables for SELENIUM_HOST and SELENIUM_PORT to localhost:4445.
CredentialsIf you have already created a credentials variable for your Sauce Labs account, use the drop-down menu to choose it as the authentication account for this project. If you have not created a credentials variable yet, click the Add button to do that now. See Creating Your Sauce Labs Credentials for details.
WebDriverChoose one or more operating system and browser combinations you wish to test using the WebDriver automation tool. If you specify one option, the plugin populates the following environment variables with values that correspond to your selection.
  • SELENIUM_PLATFORM
  • SELENIUM_BROWSER
  • SELENIUM_VERSION
  • SELENIUM_DRIVER
  • SELENIUM_DEVICE
  • SELENIUM_DEVICE_ORIENTATION

If you choose multiple options, the plugin populates the SAUCE_ONDEMAND_BROWSERS environment variable with a JSON string containing the attributes of each browser in your configuration.
AppiumThis setting is the same as the WebDriver setting, but it is appplicable for use with a mobile browser automation tool.
Native App Package PathIf the project is testing a native app, this is the directory location of the app package. This value will populate the SAUCE_NATIVE_APP environment variable for all Sauce tests in this project.
Use latest version of selected browsersAutomatically set the SELENIUM_VERSION environment variable to the latest version of the test browser.
Use latest version of Sauce ConnectAutomatically check for and use the latest version of Sauce Connect when launching a new tunnel for this project.
We recommend enabling this option because Sauce Connect releases are independent from plugin releases, so the Sauce Connect version bundled in the plugin may become out of date sooner than the plugin itself.
Clean up jobs and uniquely generated tunnels instead of waiting for timeoutsIf Create a new unique Sauce Connect tunnel per build in enabled in the Advanced Options section, checking this option ensures that cancelled builds do not tie up tunnels unnecessarily.
  1. Scroll to the Sauce Connect Advanced Options section, and click Advanced to display additional options described in the following table as needed.
    SettingDescription
    Sauce Connect Launch ConditionChoose an option from the sub-menu to specify a more granular app for when a tunnel should be launched for builds in this project. The default value is Always.
    Enable Verbose LoggingInclude Sauce Connect output in the Jenkins console output for each job.
    Launch Sauce Connect On NodeLaunch Sauce Connect on a Node instead of the Server.
    Sauce Host If you have a dedicated Sauce Connect instance running elsewhere, you can set the host here and override the default SELENIUM_HOST value (localhost when Sauce Connect is enabled or ondemand.saucelabs.com if Sauce Connect is not enabled).
    Sauce PortIf you have a dedicated Sauce Connect instance running elsewhere, you can set the port here and override the default SELENIUM_PORT value (4445 when Sauce Connect is enabled or 4444 if Sauce Connect is not enabled).
    Sauce Connect OptionsA list of command line arguments to apply each time Sauce Connect Proxy is launched for this project. For example:
    -i projectTunnel -l ./scp_project_log
    Create a new unique Sauce Connect tunnel per buildGenerates a unique tunnel identifier for each build in this project and populates a TUNNEL_NAME environment variable. You must then reference this variable in the capabilities for your tests.
    Sauce Connect Binary LocationA local path that will be the Sauce Connect binary extraction directory for this project. This value will override the default directory and the global setting.
    NOTE: Always run Sauce Connect on the same network as the site or app under test, but the same machine is not required.
    Set GitHub commit status with custom context and messageIf you are using an upstream job with the GitHub Pull Request Builder Plugin, enable this option to specify custom context and message values for the pull request in the upstream job.
    With AntAn option to run the Sauce Labs tests in this project using Apache Ant.

Populating your Capabilities with Environment Variables​

Sauce Labs tests use capabilities settings to specify the environment on which a test will run. Many of the plugin configurations you have set in the preceding section automatically generate applicable environment variables that can then be used to populate your test capabilities.

Environment Variables​

The following environment variables are relevant for Sauce Labs tests running in Jenkins and can be used to populate your test capabilities. Most of the environment variables defined here are automatically generated based on your project configurations for the plugin and Sauce Connect.

VariableDescriptionUsage
SELENIUM_HOSTIdentifies the Selenium server host.Configured by the Sauce Host project setting. When not set, defaults to localhost if Sauce Connect is enabled or ondemand.saucelabs.com if Sauce Connect is not enabled.
SELENIUM_PORTIdentifies the Selenium server port.Configured by the Sauce Port project setting. When not set, defaults to 4445 if Sauce Connect is enabled or 4444 if Sauce Connect is not enabled.
SELENIUM_PLATFORMThe operating system on which the browser being tested is installed.Populated by the WebDriver or Appium operating system combination specified during project configuration.
SELENIUM_VERSIONThe version number of the browser being tested.Populated by the WebDriver or Appium operating system combination specified during project configuration or dynamically if the Use latest version of selected browsers option is enabled.
SELENIUM_BROWSERThe name of the browser being tested.Populated by the WebDriver or Appium operating system combination specified during project configuration.
SELENIUM_DRIVERContains the operating system, version and browser name of the selected browser, in a format designed for use by the Selenium Client Factory.Populated by the WebDriver or Appium operating system combination specified during project configuration.
SELENIUM_DEVICEThe type of hardware on which the browser being tested is running.Populated by the WebDriver or Appium operating system combination specified during project configuration.
SELENIUM_DEVICE_ORIENTATIONThe direction of the device (Portrait or Landscape) used for testing.Populated by the WebDriver or Appium operating system combination specified during project configuration.
SELENIUM_URLThe initial URL to load when the test begins.Not automatically populated.
SAUCE_USERNAMEThe username of the Sauce Labs account on which tests in this project are run.Populated by the Username value of the authentication credential associated with the project.
SAUCE_ACCESS_KEYThe access key of the Sauce Labs account on which tests in this project are run.Populated by the Access Key value of the authentication credential associated with the project.
SELENIUM_STARTING_URLThe value of the Starting URL field.This value is not populated by any configuration setting.
SAUCE_ONDEMAND_BROWSERSA JSON-formatted string containing a set of attributes for multiple operating system and browser combinations.Populated when you select more than one WebDriver or Appium value during project configuration.
TUNNEL_IDENTIFIER or TUNNEL_NAME (as of v1.207)The unique tunnel identifier used when Sauce Connect is launched.Populated when the Create a new unique Sauce Connect tunnel per build option is selected during project configuration.

| JENKINS_BUILD_NUMBER | The ID of the build the Sauce OnDemand plugin will use when showing results that are not in the logs. | Populated when the buildName capability is set for the test. | | SAUCE_BUILD_NAME | The name of the build the Sauce OnDemand plugin will use when showing test results. | The plugin automatically populates this value at run-time with ${JOB_NAME}_${BUILD_NUMBER}. |

Referencing Environment Variables in Your Tests​

Sauce Labs automation test use the capabilities parameters to tell Sauce everything it needs to know before the test runs, such as what device, operating system, and browser the test should target; what versions of those systems; and what Sauce user is authorizing the test. Using environment variables as the values for the tests capabilities allows you to run a single test against multiple environments. For example, several of the following test capabilities all use the SAUCE_ON_DEMAND environment variable, which will loop through multiple platform combinations configured for the project.

desiredCapabilities.setBrowserName(System.getenv("SAUCE_ONDEMAND_BROWSERS"));
desiredCapabilities.setVersion(System.getenv("SAUCE_ONDEMAND_BROWSERS"));
desiredCapabilities.setCapability(CapabilityType.PLATFORM, System.getenv("SAUCE_ONDEMAND_BROWSERS"));
desiredCapabilities.setCapability(build, System.getenv("SAUCE_BUILD_NAME"));

Parallel Testing​

The OnDemand plugin supports automating your tests to run on many different combinations of device and browser simultaneously. To enable this capability for your Jenkins projects, set your project up as a Multi-Configuration Project.

  1. From your Jenkins dashboard, select New item to define a new project.
  2. Enter a name for your project and choose Multi-configuration project as the type, and then click OK.
  3. In the project's configuration page, go to the Configuration Matrix section and click Add Axis.
  4. Choose the Sauce OnDemand test type (WebDriver or Appium) that matches the type of tests you will run in this project. This will dictate the operating system/browser combinations available in the next step.
  5. Select all the operating systems and browser combinations that you want to test against.

When you run a build for this project, it kicks off separate jobs for each OS/browser combination you specified, populating the SELENIUM_PLATFORM, SELENIUM_VERSION, SELENIUM_BROWSER, and SELENIUM_DRIVER environment variables and running them simultaneously.

Alternatively, you can configure your project so you can choose specific operating system/browser combinations at run-time rather than configuring the combinations in the build specification itself. This option allows you to only test those environments that may be affected by a recent change, for example.

  1. From the Configure page of your project, check the This project is parameterized setting.
  2. Click the Add Parameter button and select the Sauce Labs Browsers option from the list.
  3. When you run the tests in this project, click the Build with Parameters menu item.
  4. In the parameters selection screen, choose the operating system/browser combinations you wish to test for this iteration.

Jenkins populates the SELENIUM_PLATFORM, SELENIUM_VERSION, SELENIUM_BROWSER, and SELENIUM_DRIVER environment variables for each combination you specified and runs the tests in parallel.

Reporting between Sauce Labs and Jenkins​

VIRTUAL CLOUD ONLY

The following sections describe how to share information about your Sauce Labs tests in both the Sauce Labs site and your Jenkins dashboard.

Capture Build Details​

Set the SAUCE_BUILD_NAME environment variable as the value of the build desired capability to set the Sauce build name at runtime. This enables you to access your test reports by build in Sauce Labs and view them on the Jenkins Build Details page.

Jave Build Capabilities Example
DesiredCapabilities capabilities = new DesiredCapabilities();
// ...
capabilities.setCapability("build", System.getenv("SAUCE_BUILD_NAME");

Publish Test Status to Sauce Labs​

The Sauce plugin for Jenkins will also mark the Sauce jobs as passed or failed, but you need to configure Jenkins to parse the test results.

  1. From the Configure page of your project, select the Post-Build Actions tab.
  2. Select Run Sauce Labs Test Publisher.

Output the Jenkins Session ID to stdout​

As part of the post-build activities, the Sauce plugin will parse the test result files in an attempt to associate test results with Sauce jobs. It does this by identifying lines in the stdout or stderr that have this format: SauceOnDemandSessionID=<session id> job-name=<some job name>

The session id can be obtained from the RemoteWebDriver instance and the job-name can be any string, but is generally the name of the test class being executed.

To make sure that your test results and Sauce jobs are associated properly, you need to output the session id to stdout. For example, this is the code you would use to output the session id to the Java stdout.

private void printSessionId() {

    String message = String.format("SauceOnDemandSessionID=%1$s job-name=%2$s",
    (((RemoteWebDriver) driver).getSessionId()).toString(), "some job name");
    System.out.println(message);
}

Setting up Multi-Node Job Queuing​

One of the most helpful features of Jenkins CI is automatic job queuing. If there are more build jobs requested than there are resources to execute those jobs, Jenkins can queue your tests, executing them in the order they were requested as resources become available. Or you can use labels to specify the resources you want to use for specific jobs, and set up graceful queuing for your tests.

On the Jenkins dashboard, The Build Queue and Build Executor Status panels show the Nodes' capacity for running jobs. By default, a Node with two executors can run up to two jobs at once and any additional jobs are added to the Job Queue and will run on the next executor that becomes free.

Assigned Nodes let you define Nodes for specific purposes, such as dedicated platforms, as well as for load balancing and other functions. To assign projects to a specific Node, the Node must have a label.

To label a Node and assign a project to it:

  1. From the Jenkins Dashboard, select Manage Jenkins, then click Manage Nodes & Clouds and choose Add New Node.
  2. Provide a name for the Node and the number of executors it can use.
  3. Add a descriptive Label, such as sauceJobs.
  4. From the Configure page of any project you wish to run on this Node, check Restrict where this project can be run to enable the Label Expression field, then enter the label of the applicable Node for the project.

Projects configured to run on specific Nodes queue to run on their assigned Nodes according to availability.

Using the OnDemand Plugin with the Jenkins Pipeline​

Pipeline is a plugin for Jenkins, based on the Groovy programming language, for managing your Continuous Deployment process. The OnDemand plugin lets you authenticate to Sauce Labs and manage Sauce Connect so you can take advantage of your Jenkins Pipeline integration to run your Sauce Labs tests.

Creating the Sauce Block Snippet​

The {sauce} block lets you pass your Sauce Labs username and access key as environment variables to Jenkins.

  1. Enable the Snippet Generator in Jenkins Pipeline.
  2. Select sauce: Sauce and Generate Groovy.
  3. Add the returned snippet to your Groovy script.

Creating the Sauce Connect Block Snippet​

The {sauceconnect} block lets you manage starting and stopping Sauce Connect. Wrap it with the {sauce} block so your authentication is included.

  1. Enable the Snippet Generator in Jenkins Pipeline.
  2. Select sauce: Sauce Connect and Generate Groovy.
  3. Add the returned snippet to your Groovy script within the {sauce} block.

The following example shows the sauce and sauceconnect snippets as they would be added in the Groovy script.

Sample Sauce and Sauce Connect Block Snippets

node('mac') {
    sauce('36987f5a-62da-40ac-bbc0-583806f9df4d') {
        sauceconnect(useGeneratedTunnelIdentifier: true, verboseLogging: true) {
            sh 'env | sort'
        }
    }
}

Creating the Sauce Publisher Snippet​

The {saucePublisher} function lets you send test result data to Sauce Labs. See Publishing Test Status to Sauce Labs.

  1. Enable the Snippet Generator in Jenkins Pipeline.
  2. Select saucePublisher: Run Sauce Labs Test Publisher and Generate Groovy.
  3. Add the returned snippet to your Groovy script.
note

It's not required to wrap the {saucePublisher} in the {sauce} snippet, but do include the {saucePublisher} in some part of the Pipeline file in order to report the results.