We've all accepted the Continuous Integration religion here, right?  Our Visual Studio Team System (VSTS) WebTests are part of our unit test suite and should be run as part of our continuous integration Team Builds, right?

Well, if we want our VSTS WebTests do something more than just a couple of light-hearted clicks around our ASP.NET application we're going to need some custom code.  This means writing and using custom Web Test Plugins, custom Validation Rules, and custom Extraction Rules in our web tests.

None of these things are especially difficult to write but has anyone actually tried to make these things run as part of a Team Build?  I eventually got them working with Buck Hodges' Unit TestToolsTask Power Tool but have yet to get them working .vsmdi test lists.  (Here's some more info on the TestToolsTask.)  All my tests worked great from the Visual Studio Test Runner but getting them working through Team Build was -- well -- not so simple.

When I'd go to run these tests, I'd get an error in my build saying "MSBUILD : warning : Test Run Error."  Then I'd go look in the TestResults folder at the Visual Studio Test Results Files (*.trx) and find an error saying it couldn't find the Web Test Plugin assembly.

Could not run web test 'DataDrivenPersonWebTest' on agent 'TEAMSERVER': Could not create instance of class 'PersonWebTestPlugin' :Could not load file or assembly 'WebTestResearch.WebTestPlugins' or one of its dependencies. The system cannot find the file specified.

Hmmm.  Well, that's not good.  I looked in the $(OutDir) for the build and the DLLs I wanted/needed were all right there.  So why couldn't the build find them?

(If you're not into the "how I figured it out" stuff, just skip down to The Solution.)

I tried strong-naming my assemblies and registering them in the GAC as part of the build.  No luck.

Then I went in with Reflector and started looking at how the WebTest actually creates instances of the plugin classes.  Inside of Microsoft.VisualStudio.QualityTools.WebTestFramework.dll (C:Program FilesMicrosoft Visual Studio 8Common7IDEPublicAssemblies) in the   Microsoft.VisualStudio.TestTools.WebTesting namespace, there is a class named "WebTestClassName".  This appears to be where the WebTest Plugin is instantiated at run time.  The CreateInstance() method references the WebTestPlugin via the MyType property which returns an instance of System.Type.  The MyType property is the root of the plugin instantiation problem.  It uses the Type.GetType(string, bool) method and this method does not try to load from the GAC.

There's a nifty little tool that comes with the .NET Framework called the Assembly Binding Log Viewer (aka. fuslogvw.exe).  Fuslogvw lets you configure logging in order to see where the .NET Framework is looking when it tries (and fails) to load an assembly.  Here's what I got for output:

*** Assembly Binder Log Entry  (7/5/2007 @ 11:54:42 AM) ***

The operation failed.
Bind result: hr = 0x80070002. The system cannot find the file specified.

Assembly manager loaded from:  C:WINDOWSMicrosoft.NETFrameworkv2.0.50727mscorwks.dll
Running under executable  C:Program FilesMicrosoft Visual Studio 8Common7IDEvstesthost.exe
--- A detailed error log follows.

=== Pre-bind state information ===
LOG: User = WORK0administrator
LOG: DisplayName = WebTestResearch.WebTestPlugins
(Partial)
LOG: Appbase = file:///C:/Program Files/Microsoft Visual Studio 8/Common7/IDE/
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = vstesthost.exe
Calling assembly : Microsoft.VisualStudio.QualityTools.WebTestFramework, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a.
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: C:Program FilesMicrosoft Visual Studio 8Common7IDEvstesthost.exe.config
LOG: Using machine configuration file from C:WINDOWSMicrosoft.NETFrameworkv2.0.50727configmachine.config.
LOG: Policy not being applied to reference at this time (private, custom, partial, or location-based assembly bind).
LOG: Attempting download of new URL
file:///C:/Program Files/Microsoft Visual Studio 8/Common7/IDE/WebTestResearch.WebTestPlugins.DLL.
LOG: Attempting download of new URL
file:///C:/Program Files/Microsoft Visual Studio 8/Common7/IDE/WebTestResearch.WebTestPlugins/WebTestResearch.WebTestPlugins.DLL.
LOG: Attempting download of new URL
file:///C:/Program Files/Microsoft Visual Studio 8/Common7/IDE/PrivateAssemblies/WebTestResearch.WebTestPlugins.DLL.
LOG: Attempting download of new URL
file:///C:/Program Files/Microsoft Visual Studio 8/Common7/IDE/PrivateAssemblies/WebTestResearch.WebTestPlugins/WebTestResearch.WebTestPlugins.DLL.
LOG: Attempting download of new URL
file:///C:/Program Files/Microsoft Visual Studio 8/Common7/IDE/PublicAssemblies/WebTestResearch.WebTestPlugins.DLL.
LOG: Attempting download of new URL
file:///C:/Program Files/Microsoft Visual Studio 8/Common7/IDE/PublicAssemblies/WebTestResearch.WebTestPlugins/WebTestResearch.WebTestPlugins.DLL.
LOG: Attempting download of new URL
file:///C:/Program Files/Microsoft Visual Studio 8/Common7/IDE/WebTestResearch.WebTestPlugins.EXE.
LOG: Attempting download of new URL
file:///C:/Program Files/Microsoft Visual Studio 8/Common7/IDE/WebTestResearch.WebTestPlugins/WebTestResearch.WebTestPlugins.EXE.
LOG: Attempting download of new URL
file:///C:/Program Files/Microsoft Visual Studio 8/Common7/IDE/PrivateAssemblies/WebTestResearch.WebTestPlugins.EXE.
LOG: Attempting download of new URL
file:///C:/Program Files/Microsoft Visual Studio 8/Common7/IDE/PrivateAssemblies/WebTestResearch.WebTestPlugins/WebTestResearch.WebTestPlugins.EXE.
LOG: Attempting download of new URL
file:///C:/Program Files/Microsoft Visual Studio 8/Common7/IDE/PublicAssemblies/WebTestResearch.WebTestPlugins.EXE.
LOG: Attempting download of new URL
file:///C:/Program Files/Microsoft Visual Studio 8/Common7/IDE/PublicAssemblies/WebTestResearch.WebTestPlugins/WebTestResearch.WebTestPlugins.EXE.
LOG: All probing URLs attempted and failed.

Turns out that when you run the web tests they actually run in a separate .exe from the build -- vstesthost.exe.  VsTestHost.exe was looking to load my plugin assemblies from inside of "C:/Program Files/Microsoft Visual Studio 8/Common7/IDE/".

That's the directory that I have to copy my DLLs to in order to load them.  Conveniently, this directory already exists in Team Builds as the $(TeamBuildRefPath) property.

The Solution: Copy all the files I need to $(TeamBuildRefPath) after compile but before the tests execute.  I did this by overriding the "AfterBuild" target.

It's definitely a little (read: extremely) ugly but it works.  Anyone have any better ideas on how to do this?

-Ben