Azure App Service Slot Swap is Hanging or Taking Forever

If you’re using Azure App Services to host your apps and services, there’s a feature you might not know about called deployment slots. Why do I like deployment slots? Well, they’re great for having multiple versions of you app live at the same time. For example, a test version and the production version.

In the test and prod scenario, there’s a great feature called “swap slots”. When you run the swap operation, Azure makes the “test” slot “production” and the “production” slot into “test”.

Usually, this slot swap happens in a matter of seconds but I was running into a problem where the slot swap was taking 10 or 15 minutes and sometimes even timed out and failed. Also, while the swap operation was running (and running and running and running and running), I couldn’t edit anything for the App Service through the Azure Portal. When I tried to modify settings using the Azure CLI, most of the operations would fail saying that there was a pending operation.

TL;DR? Scroll to the Summary

Why is the Slot Swap So Slow?

When I first tried to diagnose my slot swap problems, I figured it was a problem with Azure. I thought (mistakenly) that the swap was pretty much just a swap — take this and put it here and then move this thing over here. As with so many things in technology, there was more to it.

When the slot swap operation starts, the App Service tries to make sure that the code in the slot is ready to go. Another way of saying this is that it tries to make sure that your app is warmed up and ready to start accepting requests.

By default, the App Service is going to try to access your app by issuing a GET request to the root of the app (“/”) on port 80 or 8080. Notice that I’m saying 80 and 8080 and not 443. Under the surface, your application is treated as a container and the SSL/HTTPS implementation is handled by the App Service host. That means that if your app isn’t listening on 80 or 8080, then the slot swap warmup request is going to fail.

Diagnosing the Problem: Use the Log Stream

If you’re having a slot swap hang problem, how do you see what’s going on? Well, the easiest thing to do is to go to the azure portal and go to the slot that you’re trying to swap with. For example, if you’re trying to make the Test slot into the Prod slot, then you’ll look at the Test slot in the portal. Once you’re at the page in the portal for that slot, scroll down to the Monitoring section and then choose Log stream.

You’ll then see the live log stream for your App Service slot. This will show you messages from the app service itself PLUS whatever console logging that your app is doing.

If your slot swap is hanging, you’ll probably see a message at the bottom of your log that says “Initiating warmup request to container”…and then no other messages after that until the swap times out.

2022-05-24T12:19:34.449Z INFO - Starting container for site
2022-05-24T12:19:34.462Z INFO - docker run -d -p 8080:8080 --name codegen-v3__e12f_33_ebf5b3ee -e WEBSITE_AUTH_ENABLED=True -e WEBSITE_SITE_NAME=codegen-v3 -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=codegen-v3-test.azurewebsites.net -e WEBSITE_INSTANCE_ID=a53d07fbf2bfb6b036893643ca78a6219b03a1be35c85eefc627ebf2538bb625 -e HTTP_LOGGING_ENABLED=1 appsvc/dotnetcore:6.0_20220405.4 dotnet Benday.CodeGenerator.WebApi.dll
2022-05-24T12:19:44.494Z INFO - Initiating warmup request to container codegen-v3__e12f_33_ebf5b3ee for site codegen-v3__e12f
2022-05-24T12:19:46.687Z INFO - Container codegen-v3__e12f_33_ebf5b3ee for site codegen-v3__e12f initialized successfully and is ready to serve requests.
2022-05-24T12:19:46.688Z INFO - Initiating warmup request to container codegen-v3__e12f_33_ebf5b3ee_middleware for site codegen-v3__e12f

This log stream also will show you the “docker run” command that the app service is using to start your app. One of the key things that to look for is the “-p” option because that’s the option that describes how the network port on your app’s container gets hooked up to the app service host. In this case, “-p 8080:8080” means that the app service is going to listen to port 8080 on the container and use it as port 8080 on the app service host.

Is your app running on port 8080? Mine wasn’t. And that was my problem.

Slot Swap for Angular Application is Failing / Hanging / Slow

My app is an Angular application with a .NET Core 6.0 WebApi backend. My app uses the default settings from the .NET 6 angular project template. To create my project, I did a “dotnet new angular” command.

The thing that I didn’t pay close attention to is that that “dotnet new angular” template configures the WebApi app to run on port 5000 for HTTP and 5001 for HTTPS. Considering that the App Service slot warmup tries to listen for my app on port 80 or 8080…well…that just wasn’t going to work. The app was running inside of the container but the port mappings were wrong.

BTW, just as an experiment, I tried deploying my Angular/WebApi app using Visual Studio’s Publish option. It published fine and the App Service ran but when I went to hit the app service in a browser, I just got a blank screen. Once again, the container port mapping was wrong.

Another thing to keep in mind if you’re writing an Angular app is that during development, the Angular app’s service calls get pushed through a proxy server running on yet another port. If you deploy a ‘Debug’ build of your Angular app and you find that your service calls are all failing, it might have something to do with the Angular service proxy. Try deploying a ‘Release’ build of your Angular app and see if that helps.

Customize the Slot Warmup Settings: Listen on the Correct Port

I suppose that when I really think about it, my problem wasn’t exactly a problem with the Azure App Service Slot Swap. It was actually a misconfiguration of the App Service.

In order to get the App Service to listen to the proper port on the container, there’s a setting called WEBSITES_PORT. To set that value, go to the Azure Portal and choose Configuration.

Under Application settings, add a new application setting named WEBSITES_PORT and set the value to the port on your app. In my case with the Angular application running under .NET 6 WebApi, the value for WEBSITES_PORT is 5001.

Customize the Slot Warmup Settings: Other Options

There are some additional options for how the slot swap warm up happens: WEBSITE_SWAP_WARMUP_PING_PATH and WEBSITE_SWAP_WARMUP_PING_STATUSES. The ‘ping path’ value gives you the option to call a specific URL path on your application instead of just the default of calling root ‘/’. The ‘ping statuses’ value lets you specify what HTTP response code you’ll accept in order to validate that the warmup has happened.

By default, the swap warmup is perfectly content to accept any HTTP response code from your app. Did your app return an HTTP 500 error? Well, the app service warmup is happy with that. If you wanted to only allow HTTP 200, then you’d add a value in the application settings for WEBSITE_SWAP_WARMUP_PING_STATUSES with the value of 200. If you’re set a value for WEBSITE_SWAP_WARMUP_PING_STATUS, any value that isn’t in that list will cause the slot swap to abort.

Summary

If your Azure App Service deployment slot swap is taking forever, failing, or is timing out, first remember that the slot swap issues a warmup for your app. Next, start thinking about how the app service decides if your app is running. Also remember that the App Service is probably running your application as a docker container so think about how you’d debug that container deployment.

Here are some tips for diagnosing and fixing the problems:

1. Check the Log stream for the source deployment slot
2. Is the “docker run” port mapping correct?
3. If your .NET Core app isn’t running on port 80 or 8080 (NOT 443!!!), consider specifying a custom value for “WEBSITE_PORT”
4. If you want to call a custom page in your app on warmup, set a value for WEBSITE_SWAP_WARMUP_PING_PATH
5. If you want to validate the HTTP response code from your app during warmup, set a value for WEBSITE_SWAP_WARMUP_PING_STATUSES

And if you’re running an Angular app with WebApi, remember that you’re probably by default running on the wrong port and also remember that the Angular ‘debug’ build uses a proxy server for service calls. That means you’ll probably want to deploy an Angular ‘release’ build.

I hope this helps.

-Ben

— Looking for help writing for or deploying to Azure App Services? Trying to get the most out of your App Services? Need to set up a continuous integration / continuous deployment pipeline from Azure DevOps or GitHub for your App Services? We can help. Drop us a line at info@benday.com.