Testspace is used to measure and track metrics that help teams manage the quality of their software. Our goal is to provide actionable insights from test output produced during software development — Quality Analytics for the test automation workflow.

We’ve discovered through our product development and from our customers that even with today’s automated toolchains, some aspects of managing software quality can still be very manual and time-consuming.

Today, we’re excited to announce new features that significantly reduce the time and effort when triaging and managing test failures in large result sets.

Why would I use them?

As software projects grow in size and complexity, the same occurs for test output. Development teams, including ours, manage the growing content using test suites and folders and by adding levels of hierarchy as needed.

When failures occur they can be widely dispersed and deeply nested.

Until now, triaging failures required manual click-navigation, with large result sets often time-consuming and even frustrating to manage. Failing cases in test suites spanning multiple screen views were difficult to locate when scrolling in a browser.

Once found, a failed binary result status was limited to the state of the current build — without historical context.

Although critically important, the binary status failed to answer other important questions.

• Is the failure new, or did it occur in the previous build?
• And if not in the previous, what about other recent builds?
• Is it consistently failing and for how long?
• Is the test case toggling between passing and failing?
• And for all failures, how can we tell when they’ve been resolved?

Understanding and managing failures came with a cost that increased over time.

How do they work?

When viewing results, a single click on the new Failures button reduces a complex hierarchy of folders and suites into a single flat view showing only the suites that contain test failures.

When opening a test suite with failures, only the failing test cases are shown by default. A simple click removes the filter and renders all cases viewable. Another option in the failure view is to further filter based on new and/or frequency of failures. High frequency failures are cases that have failed at least three times in the last five results.

Test cases that fail are now tracked and monitored over subsequent result sets and are reported in one of three states

• New – when a test case transitions from a consistently passing state to failing.
• Tracked – after a test case fails and until the test case transitions into a resolved state.
• Resolved – either through correction, disablement, or removal as reported by subsequent test results.

The counts of test cases in each of the above three states are rolled up and listed under a new Failures column at every level in the results hierarchy. Test cases in a Tracked state display a failure frequency counter, the rate of failure over the last five results. Each test case in a Tracked state supports a new historical list view of up to the five most recent results of passed, failed, N/A, or missing.

Read more

Get setup in minutes!

Try Testspace risk-free. No credit card is required.

Have questions? Contact us.