Azure DevOps, Scrum, & .NET Software Leadership and Consulting Services

Introducing Slide Speaker: Videos with Voice-over from your PowerPoint and Google Slides Presentations

How to get accurate Processor % numbers from VMWare during a Load Test.

If you read this blog from time to time, you know that I do a fair amount with Visual Studio Load Testing.  Well, when I’m working with a customer who hosts their apps inside of VMWare, it’s always been a just pain to know what’s *really* going on with processor utilization inside the virtual machine (aka. “inside the guest OS”.)

The issue is that VMWare’s hypervisor does a lot of magic that causes the Windows PerfMon counters for the Processor to be wildly inaccurate inside of the virtual machines. 

Every once in a while, I’d search around on the internet(s) for a solution but never came up with anything and I’d pretty much written off knowing the truth about Processor % Utilization in my Visual Studio Load Tests.  Well, I got grumpy enough at VMWare today and did another search. 

THIS TIME I FOUND AN ANSWER!!!  Someone named “Vitaliy S.” in a VMWare Forum thread posted that there’s a Performance Monitor Category called “VM Processor” and “VM Memory”.  Finally!

SNAGHTMLe4e0883

I added the VM Processor “% Processor Time” counter to my load test and graphed it against the standard Windows Processor “% Processor Time”.  As you can see from the graph below, there’s a pretty significant difference between the two. 

SNAGHTMLe4905b5

BTW, the sharp-minded readers out there may also notice that VMWare thinks it’s giving this VM 107% processor utilization…so…uhhh…that’s a little weird.  Whatever, at least I’ve got some better PerfMon counters for the processor in VMWare.

-Ben

SUBSCRIBE TO THE BLOG

Recent Blog Posts

more…

SUBSCRIBE TO THE BLOG

Ben Day

7 responses to “How to get accurate Processor % numbers from VMWare during a Load Test.”

  1. Bill Wilder Avatar
    Bill Wilder

    Hey Ben – I’ve always blindly trusted the perf counters on Windows Azure virtual machines, esp since they are a central part of Diagnostics. Until I read this post, it never occurred to me that they may be distorted since they are in a VM. Any experience suggesting they are not to be fully trusted?

    Reply
    1. Ben Day Avatar
      Ben Day

      Actually, Bill — that’s a really interesting point. Why trust the processor utilization for *any* virtual machine? The part that bothered me about the VMWare stuff was that Windows thought it was one value and VMWare Studio thought that it was something entirely different. As far as I know, Hyper-V doesn’t have the same inside vs. outside view of the universe…but who know…it *might*.

      -Ben

      Reply
  2. Jame Davis Avatar
    Jame Davis

    We have a pretty large VMWare infrastructure and ran into the same issue when load testing. I had to roll my own and used the MOB interface of VMWare using the VI SDK. The application basically sat as a service with a GUI interface that would poll the VMware servers once every 5 or 10 seconds; I think that 5 seconds is the best granularity for most VMs.

    It would then create windows perf counters on the machine it was running. We ran it on the controller as that was the easiest. Of course you can run it anywhere and just add that series to your Load Test.

    Agreed you can’t fully trust the counters but it is much better than the Windows perf counters.

    When we’ve tested Hyper-V we get numbers that are within 2% – 5% of actual resource utilization.

    Reply
  3. James Davis Avatar
    James Davis

    Forgot the “s” at the end of my name. 🙂

    Reply
  4. Brad Avatar
    Brad

    I believe the reason you are seeing 107% is related to how much resources are allocated to your VM from the host.I think VMware lumps the cores together or however you are shareing so 2 cores dedicated you could have 200% utilization.

    Reply
  5. Reggie Maricle Avatar
    Reggie Maricle

    Brad hit the nail on the head.

    VM Processor % Processor time can exceed 100%. The same rules apply when using counters in the Process key, its 100 * # of CPU’s. Another tip is to find the VM-Ware Process in the Process key, and log it. My exp has shown that VM over head runs less than 2%, as a rule of thumb.

    Happy tuning 🙂

    Reply
  6. Tim Hampton Avatar
    Tim Hampton

    I’ve found that even when you use the formula of 100 * # of CPUs that I’m still getting numbers over 100% per CPU. I have a couple of web server VMs that are part of the machines under test and they have two CPUs each. I’ll get Max values over 200 (224 and 229 for each VM respectively), which makes me think that this is still an unreliable metric or that VMware is doing something under the hood to extend CPU capacity when it hits 100% per core.

    Thoughts?

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.