TFS2010 Build Agent Cannot Talk to Controller: “There was no endpoint listening at http://servername:9191/Build/v3.0/Services/Controller/5 that could accept the message”

The Problem:

I ran into a problem with a customer’s Team Foundation Server 2010 build server today. The server’s been up and running for at least 6 months and then suddenly it wouldn’t run builds anymore – the builds would start and then hang with the message “Run On Agent (waiting for build agent)”. 

When I went to look at the server configuration in Team Foundation Server Administration Console’s Build Configuration dialog, the Build Controller and Build Agents both said that they were “Ready”.  The weird thing was that the Agent icon didn’t indicate that it was ready.  (see below)

In Team Explorer’s “Manage Build Controllers” dialog, that build agent was coming up as “Unavailable”.

Back in the TFS Administration console, I went to the Build Agent Properties dialog and clicked “Test Connection”.

 

After about 60 seconds I got a “Testing connections for the build agent timed out” message box.

After this error message box popped up, we’d get a TFSBuildServiceHost error in the Event Log.

Here’s the complete text of the event log error:

Log Name:      Application
Source:        TFSBuildServiceHost
Date:          11/17/2010 3:45:43 PM
Event ID:      0
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      HQDEVUTILITY
Description:
Service ‘HQDEVUTILITY – Agent1’ had an exception:
Exception Message: There was no endpoint listening at http://hqdevutility:9191/Build/v3.0/Services/Controller/5 that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. (type EndpointNotFoundException)

Exception Stack Trace:
Server stack trace:
   at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()
   at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout)
   at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.TeamFoundation.Build.Machine.IBuildControllerService.TestConnectionFromController(String agentUri)
   at Microsoft.TeamFoundation.Build.Machine.ServiceProxies.ServiceProxy`1.<>c__DisplayClass3.<Do>b__2(T channel)
   at Microsoft.TeamFoundation.Build.Machine.ServiceProxies.ServiceProxy`1.Do[TResult](Func`2 action)
   at Microsoft.TeamFoundation.Build.Machine.BuildAgentService.<>c__DisplayClass12.<TestConnection>b__11(Object )

Inner Exception Details:

Exception Message: Unable to connect to the remote server (type WebException)
Exception Stack Trace:    at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
   at System.Net.HttpWebRequest.GetRequestStream()
   at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()

Inner Exception Details:

Exception Message: No connection could be made because the target machine actively refused it 127.0.0.1:18303 (type SocketException)

Exception Stack Trace:    at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)

The Solution:

It turned out that there was a problem with the proxy settings in Internet Explorer for the build service account.  (Thanks to Richard Hundhausen for the suggestion to check the IE proxy settings.)

1. Log in as the same user as the build service account

2. Open Internet Explorer and go to Tools –> Internet Options

3. Choose the Connections tab

4. Click the “LAN settings” button

5. You should now be on the Local Area Network (LAN) Settings dialog.  Verify that the Proxy server settings are correct.  If you aren’t using a proxy server, the “Use a proxy server for your LAN” checkbox should be UNCHECKED.

6. Restart your TFS Build Service

7. Verify that your build agent is running and that the icon indicates that the agent is available.

Summary:

What appears to have happened was that the Proxy Server settings were mistakenly and incompletely pushed to the build server via Windows Domain Group Policy.  Once the proxy settings were corrected, everything worked again.

-Ben

 

— Looking for help with your Team Foundation Server?  Problems with your TFS2010 Team Builds?  Want some coaching on Microsoft Visual Studio ALM?  Drop us at line: info@benday.com