Tag Archives: Deep Dive

Hyper-V Resource Metering

Hyper-V Resource Metering

In this video we explore the new resource metering features in Hyper-V 2012 and 2012R2. We talk about the structure and uses of resource metering and then we use PowerShell to implement resource metering on a clustered configuration.

Resource Metering Overview

Resource metering enables administrators to collect data about the resource usage of a VM or pool of VMs. This data is presented as average utilization over a time period. The time period is the time sense the metering was enabled, or sense the last metering reset. This data can be used for creating chargeback or show back models. As you will soon see, the data from resource metering is relatively simple. It is not meant to replace an enterprise performance monitoring tool such as System Center Operations Manager (SCOM). Resource metering is something that can only be configured via the PowerShell interface.

Resource metering allows you to collect information about the VM’s CPU, memory, network and storage utilization. The data collected is stored in the virtual machine’s configuration file. This means that the resource metering data will stay with the VM when it is migrated to another host or cluster node.

A hierarchy of resource pools can be created to group VMs. The groups can represent any logical collection that is meaningful in your environment. A VM’s resource may only belong to one resource group at a time. The resource group gets its totals from the data stored in the VM’s configuration files. As such, if a host has no VMs on it, its resource groups will not report any values and will be disabled. When creating a resource pool a ‘pool type’ must be defined. While it is possible to configure different names for each of the resource pool types, it may be easier to name all of the pools for a particular logical collection with the same name as we have done in the video. This makes collecting the data much easier, but is not a requirement.

PowerShell Command to Manage Resource Metering

Enable/Disable Resource Metering
Enable-VMResourceMetering
Disable-VMResourceMetering
Configure VM Resources for Metering
Set-VMProcessor
Set-VMHardDiskDrive
Set-VMMemory
Set-VMNetworkAdapter
Add-VMNetworkAdapterAcl
Remove-VMNetworkAdapterAcl
Creating/Removing Resource Pools
New-VMResourcePool
Set-VMResourcePool
Remove-VMResourcePool
Measuring VMs and Pools
Measure-VM
Measure-VMResourcePool
Reset-VMResourceMetering

Resource Metering Demo!

In the video we did a demo where we created resource pools called Pool1 on cluster HVC0 for our 3 VMs (App-1,2,3). We then gathered pool metering information from all of the nodes and displayed the aggregated data.
Here is the script that we used in the video:
ClusterResourcePoolFunctions.PS1

PowerShell Code for Resource Metering

###Demo 1 functions - Demo 1 Enable VM and Measure
###Show Cluster Resource Pool
function Get-ClusterResourcePool {
    param ($Cluster = ".",$ResourcePool = "*")
    $Nodes = (Get-ClusterNode -Cluster $Cluster).name
    Get-VMResourcePool -ComputerName $nodes -Name $ResourcePool
}
### Enable Cluster Resource Metering VM
function Enable-ClusterResourcePoolVM {
    param ($Cluster = ".", $VMFilter = "*")
    $Nodes = (Get-ClusterNode -Cluster $Cluster).name
    $VMS = Get-VM -ComputerName $Nodes -VMName $VMFilter -ErrorAction SilentlyContinue
    if($VMs){ #VMs found!
        $VMs |Enable-VMResourceMetering
    } else {
        "No Vms Match Filter"
    }
}
### Measure Cluster Resource Pool VMs
function Measure-ClusterResourcePoolVM{
    param ($Cluster = ".", $VMFilter = "*")
    $Nodes = (Get-ClusterNode -Cluster $Cluster).name
    Measure-VM -ComputerName $nodes -Name $VMFilter -ErrorAction SilentlyContinue
}

#Demo 1 Enable VM and Measure
Get-ClusterResourcePool -Cluster HVC0
Enable-ClusterResourcePoolVM -Cluster HVC0 -VMFilter App*
start-vm -ComputerName HVC0N1 -Name App-2
Get-ClusterResourcePool -Cluster HVC0
Measure-ClusterResourcePoolVM -Cluster HVC0 -VMFilter App-*
stop-vm -ComputerName HVC0N1 -Name App-2

###Demo 2 functions

###Create Cluster Resource Pool
Function New-ClusterResourcePool {
    param ($Cluster = ".", $ResourcePool = "Pool1", $StoragePath = "NoneSupplied", $SwitchName="HVSwitch")
    $Nodes = (Get-ClusterNode -Cluster $Cluster).name
    New-VMResourcePool -ComputerName $Nodes -Name $ResourcePool -ResourcePoolType Ethernet,Processor,Memory
    If ($StoragePath -eq "NoneSupplied"){$StoragePath = (Get-ClusterSharedVolume -Cluster $cluster| select -ExpandProperty sharedVolumeInfo).friendlyvolumename}
    New-VMResourcePool -ComputerName $Nodes -Name $ResourcePool -ResourcePoolType VHD -Paths $StoragePath
    Get-VMSwitch -ComputerName $Nodes -Name $SwitchName| Add-VMSwitch -ResourcePoolName $ResourcePool
}
### Set Cluster Resource Pool VM Assignment
Function Set-ClusterResourcePoolVM {
    param ($Cluster = ".",$ResourcePool="Primordial", $VMFilter = "*")
    $Nodes = (Get-ClusterNode -Cluster $Cluster).name
    $VMS = Get-VM -ComputerName $Nodes -VMName $VMFilter -ErrorAction SilentlyContinue
    if($VMs){ #VMs found!
        foreach ($VM in $VMS) {
            Write-Debug "Setting resource pool $ResourcePool on VM $($VM.name)"
            $VM|Set-VMProcessor -ResourcePoolName $ResourcePool
            $VM|Set-VMMemory -ResourcePoolName $ResourcePool
            $VM|Get-VMNetworkAdapter| Set-VMNetworkAdapter -ResourcePoolName $ResourcePool
            $VM|Get-VMNetworkAdapter| Connect-VMNetworkAdapter -UseAutomaticConnection #ENABLES LIVE MIGRATION BETWEEN HOSTS IN CLUSTER
            $VM|Get-VMHardDiskDrive| Set-VMHardDiskDrive -ResourcePoolName $ResourcePool
        }
    } else {#no vms found!
        "No VMs match filter"
    }
}
### Get Cluster Resource Pool VM Assignment
Function Get-ClusterResourcePoolVM {
    param ($Cluster = ".", $VMFilter = "*")
    $Nodes = (Get-ClusterNode -Cluster $Cluster).name
    $VMS = Get-VM -ComputerName $Nodes -VMName $VMFilter -ErrorAction SilentlyContinue
    if($VMs){ #VMs found!
        foreach ($VM in $VMS) {
            Write-Debug "Getting resource pool info for VM $($VM.name)"
            $MyObj = ""| select VM, CPU, RAM, Disk, Network
            $MyObj.VM = $VM.name
            $Myobj.CPU = $vm| Get-VMProcessor|select -ExpandProperty ResourcePoolName
            $MyObj.RAM = $vm| Get-VMMemory|select -ExpandProperty ResourcePoolName
            $MyObj.Network = ($vm| Get-VMNetworkAdapter|select -ExpandProperty PoolName|select -Unique) -join ","
            $MyObj.Disk = ($vm| Get-VMHardDiskDrive|select -ExpandProperty PoolName|select -Unique) -join ","
            $MyObj
        }
    } else {#no vms found!
        "No VMs match filter"
    }
}
### Measure Cluster Resource Pool
function Measure-ClusterResourcePool {
    param ($Cluster = ".",$ResourcePool)
    $Nodes = (Get-ClusterNode -Cluster $Cluster).name
    $Pools = Get-VMResourcePool -ComputerName $Nodes | where-Object { $_.ResourceMeteringEnabled -eq "True"}|%{$_.name}| Select-Object -Unique
    foreach ($Pool in $Pools){
        $MyObj = ""| select PoolName,  AvgCPU, AvgRAM, TotalDisk, NetworkInbound, NetworkOutbound
        $MyObj.PoolName = $Pool
        $MyObj.AvgCPU = (Measure-VMResourcePool -ComputerName $nodes -name $pool -ResourcePoolType Processor `
                             -ErrorAction SilentlyContinue|Measure-Object -sum -Property AvgCPU).sum
        $MyObj.AvgRAM = (Measure-VMResourcePool -ComputerName $nodes -name $pool -ResourcePoolType Memory -ErrorAction SilentlyContinue|Measure-Object -sum -Property AvgRAM).sum
        $MyObj.TotalDisk = (Measure-VMResourcePool -ComputerName $nodes -name $pool -ResourcePoolType VHD -ErrorAction SilentlyContinue|Measure-Object -sum -Property TotalDisk).sum
        #Networking
        $networkGroup = Measure-VMResourcePool -computername $nodes -name $pool -ResourcePoolType Ethernet -ErrorAction SilentlyContinue | select -ExpandProperty NetworkMeteredTrafficReport|Group-Object -Property direction
        $MyObj.NetworkInbound  =  ($networkGroup|?{$_.name -eq "Inbound"}|select -ExpandProperty group|Measure-Object -Property TotalTraffic -Sum).sum
        $MyObj.NetworkOutbound  =  ($networkGroup|?{$_.name -eq "Outbound"}|select -ExpandProperty group|Measure-Object -Property TotalTraffic -Sum).sum
        $MyObj
    }
}

#Demo 2 Create, assign and measure pool
New-ClusterResourcePool -Cluster HVC0 -ResourcePool Pool1
Get-ClusterResourcePool -Cluster HVC0
Get-ClusterResourcePoolVM -Cluster HVC0 -VMFilter App* | Format-Table -AutoSize
Set-ClusterResourcePoolVM -Cluster HVC0 -ResourcePool pool1 -VMFilter App*
Get-ClusterResourcePoolVM -Cluster HVC0 -VMFilter App* | Format-Table -AutoSize
Start-VM -ComputerName (Get-ClusterNode -Cluster HVC0).name -Name App*
Measure-ClusterResourcePool -Cluster HVC0 -ResourcePool pool1

###Demo 3 functions
### Reset Cluster Resource Metering VM
function Reset-ClusterResourcePoolVM {
    param ($Cluster = ".", $VMFilter = "*")
    $Nodes = (Get-ClusterNode -Cluster $Cluster).name
    $VMS = Get-VM -ComputerName $Nodes -VMName $VMFilter -ErrorAction SilentlyContinue
    if($VMs){ #VMs found!
        $VMs |Reset-VMResourceMetering
    } else {
        "No Vms Match Filter"
    }
}
### Reset Cluster Resource Metering
function Reset-ClusterResourcePool {
    param ($Cluster = ".",$ResourcePool="*")
    $Nodes = (Get-ClusterNode -Cluster $Cluster).name
    Reset-VMResourceMetering -ComputerName $nodes -ResourcePoolName $ResourcePool -ErrorAction SilentlyContinue
}

#Demo 3 Reseting resource metering
Stop-VM -ComputerName (Get-ClusterNode -Cluster HVC0).name -Name App*
measure-ClusterResourcePoolVM -Cluster HVC0 -VMFilter App*
Reset-ClusterResourcePoolVM -Cluster HVC0 -VMFilter App-2
measure-ClusterResourcePoolVM -Cluster HVC0 -VMFilter App*
Reset-ClusterResourcePool -Cluster HVC0 -ResourcePool pool1
measure-ClusterResourcePool -Cluster HVC0 -ResourcePool pool1

#Demo 4 functions
### Disable Cluster Resource Metering VM
function Disable-ClusterResourcePoolVM {
    param ($Cluster = ".", $VMFilter = "*")
    $Nodes = (Get-ClusterNode -Cluster $Cluster).name
    $VMS = Get-VM -ComputerName $Nodes -VMName $VMFilter -ErrorAction SilentlyContinue
    if($VMs){ #VMs found!
        $VMs |Disable-VMResourceMetering
    } else {
        "No Vms Match Filter"
    }
}
###Remove Cluster Resource Pool
function Remove-ClusterResourcePool {
    param ($Cluster = ".",$ResourcePool)
    $Nodes = (Get-ClusterNode -Cluster $Cluster).name
    Remove-VMResourcePool -ComputerName $Nodes -Name $ResourcePool -ResourcePoolType Ethernet,Processor,Memory,VHD
}

#Demo 4 Remove resource pools
Set-ClusterResourcePoolVM -Cluster HVC0 -ResourcePool "Primordial" -VMFilter App*
Disable-ClusterResourcePoolVM -Cluster HVC0 -VMFilter App*
Remove-ClusterResourcePool -Cluster HVC0 -ResourcePool pool1
Get-ClusterResourcePool -Cluster HVC0
Get-ClusterResourcePoolVM -Cluster HVC0 -VMFilter App*| Format-Table * -AutoSize

Resources

TechNet: Introduction to Resource Metering
MSDN: Hyper-V Resource Metering Overview

Hyper-V dVMQ

Deep Dive: Configuring dVMQ in Hyper-V

Virtual Machine Queue (VMQ) is a mechanism for mapping physical queues in a NIC to the virtual NIC in a VM partition (Parent or guest). This mapping makes the handling of network traffic more efficient. The increased efficiency results in less CPU time in the parent partition and reduced latency of network traffic. Also, without VMQ, traffic for a vSwitch on particular network interface is all handled by a single CPU core. This limits total throughput on a 10GB interface to ~2.5-4.5GBits/sec (results will depend on speed of core and nature of traffic). VMQ is especially helpful on workloads that process a large amount of traffic, such as backup or deployment servers. For dVMQ to work with RSS, the parent partition must be running Server 2012R2, otherwise RSS can not coexists with VMQ.

VMQs are a finite resource. A VMQ is allocated when a virtual machine is powered on. A queue will be assigned to each vNIC with VMQ enabled until all of the VMQs are exauted. That assignment will remain in place until the VM is powered off or migrated to another hyper-v node. If you have more vNICs in your environment than VMQs on your physical adapter then you should only enable VMQ on the vNICs that will be handling the most traffic.

Static VMQ

NovsVMQ
This image represents a Hyper-V host configured without VMQ in place. All network traffic for all the VMs is handled by a single core. With static VMQ (available in 2008R2), a VMQ is assigned to a specific CPU, and will stay on the CPU independent of workloads.

Dynamic VMQ

dVMQ
This image introduces both Dynamic Virtual Machine Queue (dVMQ) and load balancing mode for NIC teaming. These features are new to Server 2012. dVMQ is very similar to VMQ with one major difference. Dynamic VMQ will scale the number of CPU cores used to handle the VMQs across pool of CPU cores. When the network workloads are light all the dVMQs will be handled by a single CPU core, but as the network workload increases so too will the number of CPU cores used. With dVMQ in 2012 each queue can only use one CPU core at a time. Also, a vNIC can only have one VMQ assigned to it.

Sum Mode/Min Mode

In our video we recommend Hyper-V Port AND Switch Independent for a Load Balance Failover Team (LBFO) configuration on switches supporting Hyper-V workloads. This load balancing mode and teaming mode will put the vSwitch in Sum mode. This mean that we will have the sum of all the VMQs from the NICs in the LBFO team. In the case of the left image above we have 2 NICs in the team each with 2 VMQs. With the team in Sum mode we have a total of 4 VMQs to allocate to vNICs. If we use AddressHash OR Switch Dependent configuration on the team, it will be placed in Min mode. In the right image above, the same hardware now only offers 2 VMQs for vNICs. This is because inbound traffic may come in on any network interface on the team for a particular vNIC. This may be a desirable configuration if you have very few vNICs on a vSwitch (vNIC count equal or less than the fewest VMQs on any NIC in the team).

Virtual Receive Side Scaling

Server 2012R2 introduces Virtual Receive Side Scaling (vRSS). This feature works with VMQ to distribute the CPU workload of recive traffic across multiple CPU cores in the VM. This effectively eliminates the CPU core bottleneck we experience with a single vNIC. To take full advantage of this feature both the host and guest need to be 2012R2. Enabling vRSS does come with the cost of extra CPU load in the VM and parent partition. For this reason, vRSS should only be enabled on vNICs that will be exceeding 2.5GBits/Sec on a regular basis.

Base and Max CPU

Base and Max CPU properties are used to configure what CPU cores will be used by VMQ.  The base processor is the first core in the group and max is the size of the group.  For example, Hyper Threading disabled and base=2 max=4 would assign cores 2-5.  VMQ will not leverage hyper threading (HT).  If HT is enabled then only even numbered cores will be used.  For example: HT enabled, base=2 max=4 would assign even numbered cores 2-8.  When ever possible it is best to choose a base value greater than 0 (or 1 in case of HT).  Creating CPU bottlenecks on core 0 has caused performance issues in some implementations.

Requirements and Configuration for VMQ

The following are required to use VMQ:
-Server 2008R2 (Static VMQ), Server 2012(dVMQ), Server 2012R2 (dVMQ+vRSS)
-Physical NICs must support VMQ
-BelowTenGigVmqEnabled = 1 for 1GB NICs (10GB NICs are auto enabled)
Follow these steps from the video to enable VMQ
0. Enable VMQ for 1GB if required
–HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\VMSMP\Parameters\BelowTenGigVmqEnabled = 1
1. Install the latest NIC driver/firmware
2. Enable VMQ in the driver for the NIC (Process will vary by NIC model and manufacturer)
3. Determine values for Base and Max CPU based on hardware configuration
4. Assign values for Base and Max CPU
5. Configure VMs

Recommendations for VMQ/dVQM/vRSS

-Use Switch Independent + Hyper-V Port to ensure the vSwitch is in SUM mode
-Always assign a base CPU other than CPU0 to ensure best performance and resiliency
-Remember when assigning Base/Max CPU using HyperThreading only even numbered cores are used
-Multiplexor Adaptors will show Base:Max of 0:0, do not change this item
-Configure Base and Max CPU for each NIC with as little overlap as possible
-Only assign Max Processor values of 1,2,4,8
–It is ok to have max processor extend past the last CPU core or number of VMQs on the NIC

Troubleshooting VMQ

Here are a few things we have seen in the field when supporting VMQ

  • Most issues with VMQ are resolved by updating to the latest version of the NIC driver!
  • VMQ appears enabled but is showing 0 queues. This may even only impact a single port on a multiport NIC.
    • *RssOrVmqPreference = 1 Must be set on all NICs that will leverage VMQ (Follow this Link for more information)
    •  HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318\[GUID of NIC Port]\*RssOrVmqPreference = 1

If you have an issue that you have experienced in your environment not listed here let me know so I can add it to the list!

PowerShell Code to Auto Configure VMQ Base/Max Processor

ConfigureVMQ.ps1

$Teams = Get-NetLbfoTeam
$proc = Get-WmiObject -Class win32_processor
$cores = $proc| Measure-Object -Property NumberOfCores -Sum|select -ExpandProperty sum
$LPs = $proc| Measure-Object -Property NumberOfLogicalProcessors -Sum|select -ExpandProperty sum
$HT = if($cores -eq $LPs){$false}else{$true}
function SetVMQsettings ($NIC, $base,$max){
    #$nic|Set-NetAdapterVmq -BaseProcessorNumber $base -MaxProcessors $max
    Write-Host "$($nic.name):: Proc:$base, Max:$max"
}
#$LPs = 4 #testing var
#$ht = $false #testing var
foreach ($team in $teams){
	$VmqAdapters = Get-NetAdapterVmq -name ($team.members)
	#Create settings
	$VMQindex = 0
	Foreach($VmqAdapter in $VmqAdapterS){
		$VmqAdapterVMQs =$VmqAdapter.NumberOfReceiveQueues
        #$VmqAdapterVMQs = 2 #testing var
		if ($VMQindex -eq 0){#first team nic
			#base proc 1+HT and max eq to num remaining cores, num queues, whatever is less
			$base = 1+[int]$ht
			$max = ($LPs/(1+$HT)-1), $VmqAdapterVMQs|sort|select -Index 0
            SetVMQsettings -nic $VmqAdapter -base $base -max $max
           }
        else{#all other nics exclusing first team nic
            if ($VmqAdapterVMQs -gt ($LPs/(1+$HT))){ #queues exceeds core count, so just start at base+1
                $base = 1+[int]$ht
                $max = ($LPs/(1+$HT)-1), $VmqAdapterVMQs|sort|select -Index 0
                SetVMQsettings -nic $VmqAdapter -base $base -max $max
            }
            else{ #cores greater than Queues so ballancing is possible
                $StepSize = [int]((($LPs/(1+$HT))-$VmqAdapterVMQs-1)/($VmqAdapters.count-1))*$VMQindex+1
                $base = $StepSize * (1+$HT)
                $max = ($LPs/(1+$HT)-1), $VmqAdapterVMQs|sort|select -Index 0
                SetVMQsettings -nic $VmqAdapter -base $base -max $max
            }
        }
		$VMQindex++
	}
}

Resources

TechNet Networking Blog: Deep Dive VMQ Part 1, 2, 3