I was visiting a new customer yesterday who is thinking of adopting Team Foundation Server 2010. Amongst other things, they asked me for an overview of the SQL Server Reporting Services Reports –- specifically, which reports are good for Project Managers. My first thought was the Stories Overview report.
The Stories Overview report (see Figure 1) gives a high-level view of what User Stories (aka ‘work’ or ‘requirements’) are scheduled for the current Iteration, how much work has been completed on each User Story, how many hours of work is remaining for each User Story, plus information about QA testing activities such as test results and bugs.
Figure 1 – The Stories Overview Report
As I was talking through the information and the value of the Stories Overview report, one of the people in the meeting asked “What’s that ‘Test Points’ column for?” Everyone else in the meeting were apparently also thinking the same thing and chimed in “yah…what’s that column for? Is that the number of Story Points for the User Story that’s been tested? How do you figure that out?” I had to admit that, although I’d looked at this report hundreds of times, I’d never noticed that Test Points column. Mentally, I’d just filtered it out.
So, yah. What is that Test Points column for and how does it get populated?
The short answer is that it’s a measure of test complexity. Basically, it answers how many different environment configurations does each requirement have associated with it. For example, in a web application, you might need to test a feature in “Windows 7 with Internet Explorer 8”, “Windows 8 with Internet Explorer Metro”, and “Windows 7 with Chrome”. That’s 3 different configurations and therefore that feature would get a Test Points value of 3 on the Stories Overview Report.
Where & How Do You Create Test Configurations?
Test configurations are set using Microsoft Test Manager (MTM). In the Visual Studio Application Lifecycle Management (aka ‘Visual Studio ALM’ or ‘VS ALM’) world, MTM is the tool for QA Testers. Where Visual Studio 2010 is the tool for developers that helps them do all their tasks as software developers, MTM is the tool for Quality Assurance people that helps them organize and do all work when they test software.
MTM introduces the concepts of Test Plans, Test Suites and Test Cases. Test Plan is the top-level group of everything that needs to be done to test a given Iteration, Sprint, or Release. Each Test Plan has one or more Test Suites. Test Suites are the list of requirements (User Stories, Product Backlog Items) that need to be tested as part of the Test Plan. Inside of the Test Suite, each requirement has one or more Test Cases. A Test Case is the list of steps that need to be performed by a QA Tester in order to validate and verify the functionality that implements the requirement. Figure 2 shows an overview of MTM’s user interface for organizing tests in a Test Plan.
Figure 2 –- Overview of the Microsoft Test Manager (MTM) Testing Center Window
If you double-click on a Test Case you get an editor (see Figure 3) that allows you to create and edit the steps that a QA Tester needs to perform.
Figure 3 –- The Test Case editor that allows you to edit test steps
Edit the Test Configurations for a Test Plan
In the right-side panel in MTM, there’s a button labeled Configurations (see Figure 4). When you select one or more Test Case from the grid and then click this button, you can then choose which Test Configurations are associated with the Test Case.
Figure 4 –- Open the editor for the Configurations for a Test Case
After clicking the Configurations button, you’ll then see the Select Test Configurations screen (Figure 5). This screen shows you the Test Cases that you selected on the previous screen and the list of available Test Configurations for the Test Suite. You can check or uncheck the configurations for each Test Case in order to describe what is required to perform the tests.
Figure 5 –- Associate test configurations with test cases on the Select Test Configurations screen
Checking or unchecking Test Configurations and then clicking the Apply Changes button associates the Configuration with the Test Case. This is the beginning of where the Test Points value in the Stories Overview report comes from.
How Configurations Become Test Points
After you have associated your Test Configurations with the Test Plans for the Requirement (User Story, Product Backlog Item), when you click on the Requirement in MTM’s Test Plan, you’ll see something like Figure 6. Figure 6 shows you the list of Test Cases for the Requirement and one of the columns is Configurations. The Configurations column shows you the number of configurations associated with each Test Case.
Figure 6 –- Test Cases for a Requirement and the Configuration count
When you add up the number of Configurations for all the Test Cases in the Requirement, this number becomes that Requirement’s Test Points value in the Stories Overview report.
-Ben
-- Got questions about Microsoft Test Manager (MTM)? Are your Test Suites a jumbled mess? Want better ‘situational awareness’ for your project? We can help. Drop us a line at info@benday.com.
