Uploaded image for project: 'Xray for Jira Cloud'
  1. Xray for Jira Cloud
  2. XRAYCLOUD-10268

GenAI:As an Admin, I should be able to track our Xray AI usage metrics

    XporterXMLWordPrintable

Details

    • Suggestion
    • Status: New
    • Resolution: Unresolved
    • None
    • None
    • AI, Logging and Monitoring
    • None
    • UNCOVERED

    • 21

    Description

      Description 
      Currently, Jira Administrators do not have visibility into how users are leveraging AI powered Test Case generation features. Admins need the ability to track AI usage at a user level, including:

      • Which user triggered AI-assisted test generation
      • The prompt used
      • Number of test cases generated per prompt
      • Date/time of usage

      This capability is critical for governance, auditing, usage optimization, and evaluating the value of AI components.

      User friction
      Administrators and managers have no way to understand:

      • How widely AI features are being used
      • Who is relying on AI to generate test content
      • Whether prompts are appropriate and aligned with internal guidelines
      • How much AI-generated content is being created over time

      This leads to challenges in both accountability and reporting.

      Steps to reproduce (right now) / Actual Result:

      1. User opens the Xray AI Test Case creation feature.
      2. User enters a prompt.
      3. AI generates test cases.
      4. No logs or usage history is available to Jira Admins.

      Current system behavior: No audit trail, no visibility, no reporting.

      IMPACT

      What would improve if solved:

      • Better monitoring of AI adoption and ROI
      • Clear audit trail for compliance and internal reviews
      • Ability to identify improper or low-quality prompts
      • Improved team performance insights
      • Supports governance, especially in regulated industries
        Impact on stakeholders:
      • Admins: Gain full visibility and control over AI feature usage
      • QA Leads / Managers: Can assess contribution and efficiency
      • Compliance Teams: Ensure responsible usage of AI
      • Developers / Product Teams: Receive insights for improving AI features
        Current workaround:
      • Relying on self-reporting
      • No automated or reliable method
      • Cannot verify prompt quality or usage volumes
        CONTEXT & EXAMPLES:

      Concrete example:

      A QA engineer generates 50 test cases using AI from a single vague prompt.
      Another engineer uses multiple structured prompts to generate smaller, more accurate sets.
      Admins currently cannot compare these usage patterns, identify issues, or optimize workflow.

      Workaround risk:

      • No audit trail → compliance and quality risk
      • Over-reliance on AI without monitoring → potential content inaccuracies
      • Difficulty managing usage
      • Reduced transparency for team performance

      Attachments

        Activity

          People

            bernardo.cottim Bernardo Cottim
            jayanthi.murthi Jayanthi Murthi
            Votes:
            2 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 20 minutes
                20m