Uploaded image for project: 'Xray for Jira'
  1. Xray for Jira
  2. XRAY-1734

Test Parameters

    XporterXMLWordPrintable

Details

    Description

      Test parameterization is a powerful practice that allows the same test to be executed multiple times with different parameters. Parameters are similar to input values (variables) that can change with each execution.

      Without the ability to define parameterized tests, all values must be hard-coded into the specification, which makes the test static and difficult to modify. Static tests lead to a lot of redundancy since you need to define and clone the same test for the different combinations of values needed to cover different scenarios or variations of the same test.

      Parameterized Tests

      Users must be able to define parameters at the Test level in a dataset. A Parameter has the following attributes:

      • Name
      • Type: [Open, List]

      Parameters can be referenced with the notation ${NAME} on the Test Steps (native or custom text fields):

      In order to manage parameters within a Test case, users must be able to manage the dataset associated with the test.

       

      List Parameters

      List Parameters contain a predefined set of options. This is similar to a single select field. For each parameter of type List users can only set values from the predefined options.

      List Parameter Example:

       

      Name: Role
      Type: List
      Options: [Admin, User, Guest, Project Admin]
      

       

      List Parameters can be created in the following levels:

      • Global settings
      • Project settings
      • Ad hoc for each parameter

      Consequently, List Parameters can be reused across multiple parameterized Tests.

      Parameterized Preconditions

      Precondition issues can also be parameterized. 

      Parameters can not be created on preconditions directly. However, it is possible to reference parameters within the precondition definition field by name. 

      The parameters will be unfolded on the execution screen, just like Test steps. For this, the dataset associated with the execution must have the same parameters, matched by name.

      Data-driven testing

      Data-driven testing is a methodology in which a table of input and output variables is used to configure a Test case. In this case, the Test is generic in the sense that it can be executed with multiple configurations.

      Xray must support data-driven testing by providing the ability to manage and import datasets for a given parametrized Test case. The dataset input and output variables will effectively be the parameters referenced within Test steps or preconditions.

      Datasets

      Definition

      dataset is a table of input and output variables for data-driven testing.

       

      Example of a dataset with a list of email and password to validate a Login Test:

      Email Password Valid
      mfburgo@gmail.com 123456789 true
      enintend@outlook.com  qwerty false
      overbom@icloud.com password false
      kronvold@gmail.com 1234567 true
      marcs@msn.com 12345678 true
      aardo@mac.com 12345 false
      gknauss@att.net iloveyou false

       

      Xray will provide the ability to define datasets through the UI by allowing users to create columns and rows manually or import datasets from external CSV files.

       

      Datasets can be defined in the following levels/entities:

      • Test (default)
      • Test Plan > Test
      • Test Execution > Test (test run)

       

      The closest dataset to the Test Run will be used to generate the iterations effectively overriding any dataset defined in higher levels:

      Test Execution - Test (Test Run) > Test Plan - Test > Test (default)

      The Test dataset is the default. If there is the need to override or change this dataset (keeping the original dataset on the Test), users can do this at the planning or execution.

       

      Combinatorial parameters

      Xray must provide a feature to generate all combinations automatically for a dataset. When defining a dataset, users must be able to define combinatorial columns. When the test is executed, all iterations (from all combinations) will be generated automatically by Xray.

       

      In the following dataset, Valid and *Roleare combinatorial columns:

      Email Password Valid* Role*
      mfburgo@gmail.com 123456789 true Admin
      enintend@outlook.com  qwerty false User
      overbom@icloud.com password    Guest

       

      Xray Will generate the following iterations:

      Email Password Valid Role
      mfburgo@gmail.com 123456789 true Admin
      mfburgo@gmail.com 123456789 true User
      mfburgo@gmail.com 123456789 true Guest
      mfburgo@gmail.com 123456789 false Admin
      mfburgo@gmail.com 123456789 false User
      mfburgo@gmail.com 123456789 false Guest
      enintend@outlook.com qwerty true Admin
      enintend@outlook.com qwerty true User
      enintend@outlook.com qwerty true Guest
      enintend@outlook.com qwerty false Admin
      enintend@outlook.com qwerty false User
      enintend@outlook.com qwerty false Guest
      overbom@icloud.com password true Admin
      overbom@icloud.com password true User
      overbom@icloud.com password true Guest
      overbom@icloud.com password false Admin
      overbom@icloud.com password false User
      overbom@icloud.com password false Guest

       

      Given that the number of combinations can be huge, Xray must warn the user and limit the number of combinations that can be generated.

      Dataset resolution

      At the Test Run level, users will see the Test specification with the resolved configuration. The rule to choose the Test configuration is as follows:

      Given a Test Run:

      If there is a TC defined in the Test Run (Test Execution individual context), then this TC must be used

      Else if there is a TC defined in the Test Plan (individual context), then this TC must be used

      Else use the default TC defined in the Test, if any.

      Test Run -> Test Plan Test → Test

       

      When executing a parametrized Test, the execution screen must provide the Test Run with the resolved dataset.

      If a parameter can not be resolved, the parameter value must be displayed with ${parameter name}.

      Execution

      When executing a parametrized Test, the execution screen must provide the Test Run with the resolved dataset.

       

      In the case of a data-driven Test, users must be able to execute all unfolded Tests for each dataset row on the execution screen.

      The iterations must be displayed along with their status within a panel named "Iterations". This panel is expanded by default. However, individual iterations are collapsed by default. Expanding an iteration will reveal all the steps with the parameter names replaced by the iteration values. 

      Users must be able to execute iterations directly. When an iteration is executed, the overall test run status must also be updated. 

      The new iterations panel must include a progress bar to display the status for all iterations.

       

      For test runs that only contain a single iteration, the iterations panel must not be displayed. Instead, users must be able to view the steps already expanded, and a new panel called "Parameters" must be displayed with the parameters and their values.

       

      The overall execution status for a data-driven Test Run will be calculated based on the partial statuses for each iteration (table row) result using the same rules implemented already by Xray. This means that if one iteration result is FAILED, then the aggregated status for Test Run will also be FAILED.

       

      How changes to datasets affect existing executions

      When a Test Run is created, the dataset is resolved and copied to the Test Run entity. Hence, changes to datasets will not affect existing Test Runs. This allows past executions to keep an accurate record of the exact specification and the dataset that was executed.

      Reporting

      Although Xray provides the aggregated result at the Test Run level, considering the results from all iterations, Xray must provide progress bars to display the status of the inner progress of Test Runs (just like with Test Steps). Users will see how many iterations are passing, failing, or any other status configured in Xray.

       

      It will not be possible to calculate the TestRunStatus or Requirement Status based on parameter values. For this Xray already provides Test Environments. These work just like Test Parameters and they can be analyzed in TestRunStatus or Requirement Status.

       

      Xray must also provide progress bars that can give the status of iterations within Test Runs on the Test Runs List report and gadget.

      Attachments

        Issue Links

          Activity

            People

              bmpc Bruno Conde
              dpca Diamantino Campos
              Votes:
              93 Vote for this issue
              Watchers:
              70 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: