Ok. Microsoft Test Manager (MTM) isn’t going away but you might have noticed that it’s kind of been put out to pasture. There were some good features – some core features – in MTM that were kind of helpful. So what do you we do now? If you’re on Team Foundation Server 2015 (TFS2015) what’s missing? What’s your strategy for the future?
How did we get here?
Microsoft Test Manager (MTM) is the “rich client” for QA and Manual Testing that hooks in to TFS. It’s been around for a long time and while it’s a great tool, it’s infamous for having a difficult and confusing user interface mostly because it had two missions that were only loosely related. Mission #1: test case planning, management, & execution. Mission #2: virtual test lab management. The world has changed a lot since MTM came out – TFS is no longer laser-focused on Windows-only development and cloud computing is huge. Test case management is pretty much still the same but what we’re testing is a lot more varied and cross-platform. How we manage our virtual test labs is vastly different now. MTM was written before Microsoft Azure or Amazon took off in the cloud space. Put those two huge changes together and it’s a no-brainer that Microsoft had to take their QA/Manual testing tools in a different direction.
What’s changed?
Before we start discussing what’s changed and what’s gone/going away, I just want to say that life is better now and the tooling is so MUCH better now. The biggest change is that pretty much everything’s moved to the web and you no longer have to fight your way through MTM’s nonsensical user interface design. [insert the sound of rejoicing crowds here] The web interface makes sense. There’s no longer an artificial separation between planning your tests, running your tests, and tracking the results – everything is in one, unified interface. Want to plan a test? Create one or just start editing an existing one. Want to run a test? Click the Run button. Want to see the status of a test case? It’s right there in the user interface.
The second biggest change is that Lab Management is done. The build system that was at the core of Lab Management has changed. (NOTE: those XAML-based builds are still there but there’s a new build system that’s a lot easier to use.) The deployment features of Lab Management have been moved in to TFS Release Management. TFS Build vNext and TFS Release Management work in concert to get stuff deployed on to servers and the integration with the QA tools are now entirely optional. Trust me…you won’t miss Lab Management.
You’ve now got two options for running tests: 1) run test cases using the web interface and 2) run test cases using the MTM client. At this point you’re thinking, “didn’t you just say that we’re pretty much done with MTM?” Short answer: yes. Longer answer: you’re done with using MTM to plan and track your tests but you still have the option to run tests using the MTM rich test client. This makes sense because the really handy stuff from the MTM Test Runner was the ability to create really descriptive bugs that contained screenshots and video recordings. Microsoft split that test execution functionality into its own EXE so you can still do all that great stuff.
Using the MTM Test Runner also lets you continue to do lightweight test automations using MTM Action Recordings. The idea with Action Recordings is that a non-technical QA person would be able to create automated tests that help speed up the QA testing process and eliminate QA testing tedium. That Action Recordings feature makes for some jaw-droppingly awesome demos. In real life, it’s still pretty limiting and brittle. It only works with certain types of Windows application technologies and for web apps it only works with Internet Explorer.
If you want to do more free-form exploratory testing, there’s a new Exploratory Testing client for Chrome. It’s pretty good and it gives you the ability to do screenshots and video recording, too. Last time that I checked, it was still in beta and as such it still had some sharp edges. Microsoft is definitely going in a good direction with that plugin though.
What’s missing?
The biggest thing that’s missing from the web interface is Test Configurations. Go ahead and try this out. Go into the test hub and create a couple of Test Cases and then look in the Configuration column. The Configuration column says “Windows 8”, right? Why the heck does it say Windows 8? I’m not testing on Windows 8…so how do I change that? Well, through the web interface, you can’t. If you want to edit Test Configurations, you need to open Microsoft Test Manager. (Grrrr.)
Do you really even care about this? The answer is a definitely maybe. If you’re just testing your application and you don’t really care about what browser or operating system or whatever miscellaneous details about the environment, then you don’t care about Test Configurations and you can completely ignore that “Windows 8” thing. If you’re more thorough in your QA testing and you need to track behaviors of the application on various operating systems and browsers, then you care a lot.
The ability to edit Test Configurations via the web appears to be coming in the next version of the product. For now, you’ll still need to pop into Microsoft Test Manager for that.
If you’re doing video recordings of yourself running your tests using the MTM test runner client, you’ll need to configure those settings in Microsoft Test Manager, too.
Another thing that’s missing (more like deprecated) is the Associated Automation feature. That used to let you hook a Visual Studio Coded UI test directly to a Test Case and then have it run as part of a Lab Management build. Since Lab Management is pretty much done, this means that Associated Automations are just about done. I wouldn’t sweat this too badly though because running just one Coded UI test is kind of a weird choice in the first place and it makes a lot more sense to just hook your entire Coded UI Test suite in to your vNext Build or Release Management flow. If you do that, the results of those tests aren’t tied to a specific Test Case but you still get to see the results in the Test Hub’s “Runs” tab.
Summary
In summary, just relax. Everything’s going to be fine. In fact, everything’s already fine and life is already so much better by not having to live in Microsoft Test Manager anymore. Plus, in the next version of the product, Microsoft’s goal is to bring us even closer to “feature parity” between MTM and the web. You have pretty much all the same features that you know and love from MTM but they’ve just been cleaned up and organized. If you’re looking for something, the answer is that it’s probably moved to the web.
-Ben
-- Got questions about where to take your TFS QA testing strategy? Need help getting going with Agile Testing with TFS? Want to better join your QA testing effort into your overall DevOps pipeline? We can help. Drop us a line at info@benday.com.
