Update Traceability Report status logic so that stories with Passed Tests or Failed tests having linked defects are marked as VERIFIED, stories with untracked failures or incomplete executions are marked as UNVERIFIED, and NOTCOVERED remains unchanged.

Description

The Traceability Report currently classifies requirements/stories into the status categories OK, NOK, NOTRUN, and NOTCOVERED, based on Test coverage and execution results.

We would like to enhance the report logic by introducing more meaningful validation statuses that better reflect defect management practices.

Specifically, stories associated with failed test executions that have a linked defect should be considered validated rather than treated as failed coverage.

The Traceability Report should provide the following status categories:

VERIFIED – Stories whose associated Tests are:

PASSED, or
FAILED, provided a defect is linked to the Test Run or any Test Run Step.

UNVERIFIED – Stories whose associated tests have one or more executions with the following statuses:

TO DO
EXECUTING
ABORTED
FAILED (without a linked defect)

NOTCOVERED – Unchanged from the current report behavior.

The objective is to distinguish known and tracked failures from unverified requirements, providing a more accurate representation of testing and quality status.

User Friction

Organizations using the Traceability Report to assess requirement verification encounter challenges such as:

• Failed Test Executions being reported as not validated even when a defect has been identified and formally linked.
• Difficulty distinguishing between known defects and uninvestigated failures.
• Reduced visibility into requirements that have been actively tested and analyzed.
• Additional manual effort to review linked defects before determining verification status.
• Potential misinterpretation of project quality and testing progress.

Steps to Reproduce
1. Create a Story/Requirement.
2. Link one or more Tests to the Story.
3. Execute the Tests and generate the following scenarios:

Scenario A

Test Execution Status = PASSED

Scenario B

Test Execution Status = FAILED
Defect linked to the Test Run or one of its Steps

Scenario C

Test Execution Status = FAILED
No defect linked

Scenario D

Test Execution Status = TO DO, EXECUTING, or ABORTED

4. Generate the Traceability Report.

Actual Result

The report categorizes stories using the existing statuses (OK, NOK, NOTRUN, and NOTCOVERED), where failed executions are not differentiated based on whether a defect is linked.

Stories with failed executions and linked defects are not treated as verified despite having an identified and tracked issue.

IMPACT
What Would Improve if Solved

• More accurate representation of requirement verification status.
• Clear distinction between known defects and uninvestigated test failures.
• Improved traceability reporting.
• Better visibility of testing progress and quality readiness.
• Reduced manual review effort when assessing requirement status.
• More meaningful reporting for release and go-live decisions.
• Improved audit and compliance reporting.

Impact on Stakeholders

Quality Engineering Teams

• Better visibility into requirements that have been validated through testing, even when defects have been identified and documented.
  Faster assessment of testing completeness.

Product Owners and Business Teams

• Clearer understanding of which requirements have been verified versus those still requiring investigation or execution.

Program and Project Managers

• More accurate reporting of delivery readiness and quality metrics.
• Improved visibility into risks and defect-tracked failures.

Compliance and Audit Teams

• Easier identification of requirements that have undergone testing and have documented defect evidence.

Xray Administrators and Report Consumers

• Reduced need for manual interpretation of execution statuses and defect linkages.

Current Workaround

Organizations typically rely on one or more of the following:

• Manual review of failed test executions.
• Verification of linked defects at the Test Run or Test Step level.
• Exporting traceability data and applying custom reporting logic.

Creating external dashboards or reports to distinguish tracked failures from unverified requirements.

CONTEXT & EXAMPLES

Many organizations consider a failed test with a documented defect to represent a verified requirement because:

• The requirement has been executed and evaluated.
• The failure has been analyzed.
• The issue has been formally tracked through the defect management process.

Example 1

• Story A
• Linked Tests:
• Test 1 → PASSED
• Test 2 → FAILED with linked defect (BUG-123)

Desired Outcome:

• Story A appears under VERIFIED.

Example 2

• Story B
• Linked Tests:
• Test 1 → FAILED
  No linked defect

Desired Outcome:

• Story B appears under UNVERIFIED.

Example 3

• Story C
• Linked Tests:
• Test 1 → EXECUTING

Desired Outcome:

• Story C appears under UNVERIFIED.
• Example 4
• Story D
• No linked tests.

Desired Outcome:

• Story D appears under NOTCOVERED.

Workaround Risk

Current workarounds introduce risks including:

• Inconsistent interpretation of testing results.
• Incorrect reporting of requirement verification status.
• Additional manual effort and review overhead.
• Reduced confidence in traceability metrics.
• Potential inaccuracies in readiness and compliance assessments.
• Increased dependency on custom reporting solutions.