Ability to define variables in the Test specification (e.g. the steps) that can be replaced by values inherited from different sources (a calling test for example). In that scenario, the called test is used as a parameterized template (e.g. "login as the specified user"). For more details on modular tests, please check the next topic.

Parameterized Tests

Users must be able to define parameters at the Test level. A Parameter has the following attributes:

• Name
• Type: [Open, List]
• Default Value

Parameters can be defined in all Xray Test types: (Manual, Generic and BDD).

Parameters can be referenced with the notation ${NAME} and on the following Xray fields:

• Test Step fields (native or custom Text fields)
• Generic Test Definition
• BDD definition

In order to manage parameters within a Test case, Xray must provide a dedicated web panel within the issue.

List Parameters

List Parameters contain a predefined set of options. This is similar to a single select field. For each parameter of type List users can only set values from the predefined options.

List Parameter Example:

Name: Browser
Type: List
Options: [Chrome, Firefox, Edge , Safari]

List Parameters are always created at the project level. Hence, these List Parameters can be reused across multiple parameterized Tests.

New List Parameters can be defined on the project parameterization settings page or on the Tests direclty.

Parameterized Preconditions

Precondition issues can also be parameterized. All Precondition types are supported as well.

There will not be a dedicated web panel within the issue. However, it is possible to reference parameters within the Precondition specification by name. 

The parameters will be unfolded on the execution screen, just like Test cases. For this, the Test case must have the same Parameters, matched by name.

Preconditions will also support data-driven testing with Test Configurations defined at the Test, Test Plan, or Test Execution level. Please check Test Configuration Scopes.

Data-driven testing

Data-driven testing is a methodology in which a table of input and output variables is used to configure a Test case. In this case, the Test case is generic in the sense that it can be executed with multiple configurations.

Xray must support data-driven testing by providing the ability to manage and import data sets for a given parametrized Test case. The data set input and output variables will be mapped to parameters defined in Test cases.

Data Sets

Definition

A Data Set is a table of input and output variables for data-driven testing.

Example of a Data Set with a list of email and password to validate a Login Test:

Email	Password	Valid
mfburgo@gmail.com	123456789	true
enintend@outlook.com	qwerty	false
overbom@icloud.com	password	false
kronvold@gmail.com	1234567	true
marcs@msn.com	12345678	true
aardo@mac.com	12345	false
gknauss@att.net	iloveyou	false

Xray will provide the ability to define data sets through the UI by allowing users to create columns and rows manually or import data sets from external CSV files.

There will be a dedicated project settings page to manage all data sets for a given project. A data set can be used by different Tests. Data Sets can also be defined at the Test level. In this case, other Test cases won't be able to reuse the same Data Set.

Permutation Data Sets

Xray must provide a feature to generate all permutations automatically for a Data Set. When defining Data Set, users must also be able to define permutation columns in any position. When the Test Configuration is unfolded, all permutations will be generated automatically by Xray.

In the following Data Set, Browser and OS are permutation columns:

Email	Password	Valid	Browser*	OS*
mfburgo@gmail.com	123456789	true	Chrome	macOS
enintend@outlook.com	qwerty	false	Firefox	Windows
overbom@icloud.com	password	false
kronvold@gmail.com	1234567	true

Xray Will generate the following Data Set:

Email	Password	Valid	Browser*	OS*
mfburgo@gmail.com	123456789	true	Chrome	macOS
mfburgo@gmail.com	123456789	true	Chrome	Windows
mfburgo@gmail.com	123456789	true	Firefox	macOS
mfburgo@gmail.com	123456789	true	Firefox	Windows
enintend@outlook.com	qwerty	false	Chrome	macOS
enintend@outlook.com	qwerty	false	Chrome	Windows
enintend@outlook.com	qwerty	false	Firefox	macOS
enintend@outlook.com	qwerty	false	Firefox	Windows
overbom@icloud.com	password	false	Chrome	macOS
overbom@icloud.com	password	false	Chrome	Windows
overbom@icloud.com	password	false	Firefox	macOS
overbom@icloud.com	password	false	Firefox	Windows
kronvold@gmail.com	1234567	true	Chrome	macOS
kronvold@gmail.com	1234567	true	Chrome	Windows
kronvold@gmail.com	1234567	true	Firefox	macOS
kronvold@gmail.com	1234567	true	Firefox	Windows

Given that the number of combinations can be very big, Xray must warn or limit the number of permutations that can be generated.

Test Configuration Scopes

Definition

A Test Configuration (TC) is:

• a single set of parameter values
• a data set (in case of data-driven tests)

Therefore, a Test Configuration (TC) is composed of one or more Test Configuration Value (TCV).

A Test Configuration can have parameter values (TCV) for parameterized Tests and Preconditions.

Test Configurations can be defined in different scopes. Xray must provide a new dialog so that users can create and override (see below) Test Configurations. Within this dialog, Test Configuration Values can be defined, edited and deleted. This is true for both data-driven tests and TCs with a single set of parameter values.

Test Scope

At the Test level, the default value can be defined for parameters. In the case of a data-driven Test, besides default values, users will also be able to define a data set for the Test.

Test Plan Scope

At the Test Plan level, users might need to define different Test configurations from what was defined on the Tests directly. There are two contexts where Test Configurations can be defined:

• Global context (Test Plan level): affects all Tests in the Test Plan.
• Individual context: for each Test in the Test Plan.

Global

Global TCs will only be applied to Tests that contain the exact same parameters as the TC.

A new button must be provided next to "filters" so that users can define a global TC for a Test Plan. The button color must be according to if there is a TC already created or not. Clicking the button will open the Test Configuration dialog.

Individual

Individual TCs will override Global TCs.

The datatable with Tests within the Tests web panel of a Test Plan issue must feature a new static column (located to the left of the current actions column) so that users can define a new configuration for each Test. This column must display a button with an icon that changes based on the presence of a Test configuration at this level.

The button must only be displayed for Tests that have parametrization defined (parameters defined at the Test level).

Test Execution Scope

At the Test Execution level, users might also need to define different Test configurations from what was defined on the Tests or Test Plans. These configurations will override the default Test and Test Plan configurations. There are two contexts where Test Configurations can be defined:

• Global context (Test Execution level): affects all Tests in the Test Execution.
• Individual context: for each Test in the Test Execution.

Global

Global TCs will only be applied to Tests that contain the exact same parameters as the TC.

A new button must be provided next to "filters" so that users can define a global TC for a Test Plan. The button color must be according to if there is a TC already created or not. Clicking the button will open the Test Configuration dialog.

Individual

Individual TCs will override Global TCs.

The datatable with Test Runs within the Tests web panel of a Test Execution must feature a new static column (located to the left of the current actions or execute column) so that users can define a new configuration for each Test. This column must display a button with an icon that changes based on the presence of a Test configuration at this level.

The button must only be displayed for Tests that have parametrization defined (parameters defined at the Test level).

Test Configuration resolution

At the Test Run level, users will see the Test specification with the resolved configuration. The rule to choose the Test configuration is as follows:

Given a Test Run:

If there is a TC defined in the Test Run (Test Execution individual context) , then this TC must be used

Else if there is a TC defined in the Test Execution (global context), then this TC must be used

Else if there is a TC defined in the Test Plan (individual context), then this TC must be used

Else if there is a TC defined in the Test Plan (global context), then this TC must be used

Else use the default TC defined in the Test, if any.

Test Run -> Test Execution (global) > Test Plan Test  -> Test Plan (global) → Test

When executing a parametrized Test, the execution screen must provide the Test Run with the resolved Test configuration.

In case a parameter can not be resolved, and there is no default value, the parameter value must be displayed like ${parameter name}.

Execution

When executing a parametrized Test, the execution screen must provide the Test Run with the resolved Test configuration.

In the case of a data-driven Test, users must be able to execute all unfolded Tests for each TCV (table row) on the execution screen.

A new section for the TCVS must be provided above the Pre-condition section called "Test Configuration". Here, users can choose which TCV (table row) they want to execute or see the results. This will be a master-detail pattern, where the "master" is the new section and the "detail" will be the test specification resolved with the TCV selected on the master.

The new test configuration section must include a progress bar to display the status for all TCVs.

The overall execution status for a Test Run will be calculated based on the partial statuses for each TCV (table row) result using the same rules implemented already by Xray. This means that if one TCV result is FAILED, then the aggregated status for Test Run will be FAILED also.

This is valid for Manual and Generic Tests.

Parameterized BDD Tests

In the case of BDD Gherkin Tests with parameters defined on the Test case:

• When there is only one TCV, all parameter values must be replaced on the scenario within the execution page or when exporting features.
• When there are multiple TCVs (data-driven Test Configuration), the Test must be a Scenario outline and the TCVs must be unfolded onto the Examples clause. This is true when executing the scenario from Jira and also when exporting features.

In the case of a data-driven BDD test (scenario outline) with an existing Examples clause in the original specification, the existing Examples must be completely overridden in case there is a Test Configuration to apply.

How changes to Test Configurations affect existing executions

When a Test Run is created, the Test Configuration is resolved and copied to the Test Run entity. Hence, changes to the Test default parameter values or changes to the Test configurations on the Test Plan will not be copied to the Test Run. This allows past executions to keep an accurate record of the exact specification and Test Configuration that was really executed.

Although the Test configuration is copied automatically to the Test Run, users can still change the TC at the Test Run level by changing the TC on the Test Execution or directly on the Test Run.

Xray must also provide an option to revert the TC to the Test or Test Plan level if needed.

Reporting

Although Xray provides the aggregated result at the Test Run level, considering all TCV results, Xray must provide progress bars to display the status for of TCV results. Users will be able to see how many Test Configuration Results are passing, failing or in any status configured in Xray.

It will not be possible to calculate the TestRunStatus or Requirement Status based on parameter values. For this Xray already provides Test Environments. These work just like Test Parameters and they can be analyzed in TestRunStatus or Requirement Status.